Oct 30 02:38:04.202907 ip-10-0-134-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 02:38:04.202919 ip-10-0-134-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 02:38:04.202927 ip-10-0-134-164 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 02:38:04.203205 ip-10-0-134-164 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 02:38:15.534993 ip-10-0-134-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 02:38:15.535013 ip-10-0-134-164 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 84f01230c0f349bb85b0abf9e56c3faa --
Oct 30 02:40:00.828094 ip-10-0-134-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:01.274419 ip-10-0-134-164 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:01.274419 ip-10-0-134-164 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:01.274419 ip-10-0-134-164 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:01.274419 ip-10-0-134-164 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:01.277982 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277897    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.277982 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277981    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.277982 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277986    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277989    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277992    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277995    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.277998    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278002    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278008    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278012    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278015    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278018    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278021    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278024    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278027    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278030    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278033    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278035    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278038    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278040    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278043    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.278079 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278045    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278048    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278051    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278053    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278056    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278058    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278068    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278071    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278075    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278078    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278080    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278083    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278086    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278089    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278092    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278094    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278097    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278100    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278102    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.278506 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278105    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278108    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278110    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278113    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278116    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278119    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278121    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278124    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278127    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278130    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278132    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278135    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278137    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278140    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278142    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278146    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278149    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278151    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278154    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278157    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.279047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278159    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278162    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278165    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278167    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278169    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278172    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278175    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278177    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278180    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278182    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278185    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278187    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278190    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278194    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278197    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278200    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278203    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278206    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278208    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.279517 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278211    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278215    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278220    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278223    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278225    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278228    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278231    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278234    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278237    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278239    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278242    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278245    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278247    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278250    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278252    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278255    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278257    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278260    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278263    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.279992 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278265    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278267    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278270    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278272    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278275    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278278    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.280457 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.278280    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.282031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282017    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.282031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282030    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282034    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282038    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282041    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282043    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282046    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282050    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282055    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282059    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282062    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282065    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282068    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282071    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282073    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282076    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282078    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282081    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282083    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282086    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.282089 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282088    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282091    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282094    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282097    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282100    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282103    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282106    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282108    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282111    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282113    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282116    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282118    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282121    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282123    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282126    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282129    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282131    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282133    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282136    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282138    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.282573 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282142    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282144    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282147    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282149    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282151    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282154    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282156    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282159    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282161    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282165    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282168    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282171    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282173    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282176    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282179    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282181    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282183    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282186    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282188    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282191    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.283047 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282193    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282196    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282198    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282201    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282203    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282206    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282208    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282211    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282213    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282216    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282219    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282221    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282224    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282227    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282230    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282232    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282235    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282237    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282240    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282242    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.283522 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282245    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282248    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282251    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282253    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282257    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282261    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282264    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282266    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282269    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282271    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282274    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282276    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282279    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282281    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282284    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282286    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282289    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282291    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282294    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.284028 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282296    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282299    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282302    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282305    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282307    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282310    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282312    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282805    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282810    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282814    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282832    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282838    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282843    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282848    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282851    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282854    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282858    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282860    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282863    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.284472 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282866    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282870    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282873    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282876    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282879    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282882    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282884    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282887    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282890    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282892    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282895    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282897    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282900    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282903    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282905    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282908    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282910    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282913    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282916    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.284947 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282919    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282922    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282925    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282928    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282931    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282934    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282937    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282939    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282942    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282944    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282947    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282950    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282952    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282955    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282957    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282959    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282962    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282964    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282967    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282970    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.285409 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282972    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282974    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282977    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282980    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282982    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282987    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282990    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282992    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282995    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.282997    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283000    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283003    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283006    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283009    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283012    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283014    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283017    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283019    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283022    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.285907 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283025    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283027    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283030    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283032    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283035    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283037    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283040    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283042    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283045    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283047    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283050    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283052    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283055    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283058    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283060    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283063    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283065    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283068    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283070    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.286372 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283073    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283075    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283078    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283080    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283083    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283086    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283088    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283091    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283095    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283097    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283100    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283102    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283105    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283107    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283110    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.283112    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284113    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284125    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284131    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284136    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284141    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:01.286861 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284145    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284149    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284154    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284157    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284160    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284164    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284167    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284170    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284174    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284177    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284180    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284183    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284186    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284191    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284198    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284201    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284205    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284209    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284213    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284216    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284220    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284223    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284227    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284229    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:01.287350 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284233    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284236    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284239    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284244    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284247    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284250    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284253    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284256    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284259    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284264    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284268    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284270    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284274    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284277    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284282    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284285    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284288    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284291    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284294    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284297    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284300    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284303    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284307    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284310    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284313    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:01.287925 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284317    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284321    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284324    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284327    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284331    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284334    2579 flags.go:64] FLAG: --help="false"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284337    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284340    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284344    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284346    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284350    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284353    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284356    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284359    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284362    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284365    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284368    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284372    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284375    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284378    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284380    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284384    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284386    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284389    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:01.288523 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284392    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284395    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284398    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284404    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284407    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284411    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284415    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284418    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284421    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284424    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284427    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284432    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284435    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284439    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284443    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284446    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284449    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284452    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284455    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284458    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284461    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284469    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284473    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284476    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284479    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:01.289136 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284482    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284485    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284489    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284491    2579 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284494    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284497    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0451b8a459baa5f0b"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284501    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284504    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284507    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284509    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284517    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284520    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284524    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284527    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284531    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284535    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284538    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284541    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284544    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284547    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284550    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284554    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284557    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284560    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284564    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284567    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:01.289731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284570    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284573    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284577    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284580    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284583    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284586    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284589    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284593    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284598    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284601    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284604    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284609    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284612    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284615    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284618    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284621    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284625    2579 flags.go:64] FLAG: --v="2"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284630    2579 flags.go:64] FLAG: --version="false"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284634    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284639    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.284643    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284781    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284784    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284788    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.290389 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284791    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284794    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284797    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284800    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284803    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284806    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284808    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284811    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284814    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284829    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284832    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284834    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284837    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284840    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284852    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284855    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284858    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284860    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284863    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284866    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284868    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.290955 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284871    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284873    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284876    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284879    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284882    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284884    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284887    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284889    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284892    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284895    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284897    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284900    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284902    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284905    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284908    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284910    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284913    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284916    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284918    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.291451 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284921    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284924    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284926    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284929    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284931    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284934    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284936    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284939    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284941    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284943    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284946    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284949    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284951    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284956    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284958    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284961    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284964    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284966    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284969    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.291923 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284971    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284974    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284976    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284979    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284982    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284987    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284990    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284992    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284995    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.284997    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285000    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285003    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285005    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285008    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285011    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285013    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285017    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285020    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.292362 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285022    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285025    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285028    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285030    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285034    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285038    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285042    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285048    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285051    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285054    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285057    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285060    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285063    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285066    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285068    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285071    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285073    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285076    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285079    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.292813 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285081    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285084    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285087    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285089    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285092    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285094    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.285123    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285160    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285164    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285168    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285171    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285176    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285179    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285182    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285187    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.293296 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285190    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285193    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285196    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285198    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285201    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285205    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285208    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285211    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285213    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285217    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285219    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285222    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285224    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285227    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285230    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285232    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285236    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285240    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285244    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.293678 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285247    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285250    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285253    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285256    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285262    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285265    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285268    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285272    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285274    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285277    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285280    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285282    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285285    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285287    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285290    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285292    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285295    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285299    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285302    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.294149 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285306    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285309    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285311    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285314    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285316    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285319    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285322    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285324    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285327    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285329    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285332    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285334    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285337    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285340    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285342    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285344    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285347    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285350    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285353    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285355    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285359    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.294626 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285362    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285365    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285367    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285370    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285372    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285375    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285377    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285380    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285382    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285386    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285389    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285392    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285395    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285397    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285400    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285403    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285405    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285408    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285410    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.295137 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285413    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285416    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285419    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285422    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285424    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285427    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285429    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285433    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285436    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285438    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285441    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285444    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285448    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285450    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285453    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285455    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285458    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.285460    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.293204    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.293222    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:01.295607 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.294014    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.294019    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294084    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294089    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294093    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294096    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294100    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294104    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294107    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294110    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294113    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294116    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294118    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294121    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294124    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294126    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294129    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294131    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294134    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.296100 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294137    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294139    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294142    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294144    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294147    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294150    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294152    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294156    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294158    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294161    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294164    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294166    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294176    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294179    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294182    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294187    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294190    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294192    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.296559 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294195    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294197    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294200    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294205    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294208    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294210    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294213    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294215    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294218    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294220    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294223    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294226    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294228    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294231    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294233    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294236    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294239    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294241    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.297106 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294244    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294246    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294249    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294251    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294254    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294256    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294260    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294268    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294271    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294274    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294277    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294280    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294282    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294285    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294287    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294290    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294293    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294295    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294298    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.297552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294300    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294303    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294306    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294309    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294311    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294314    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294317    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294320    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294322    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294325    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294327    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294330    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294332    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294335    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294337    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294340    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294342    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294345    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294348    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294352    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.298003 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294362    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294365    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294368    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294370    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294373    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294376    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294378    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294381    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294383    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294386    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294390    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.294420    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294604    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294609    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294612    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.298689 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294614    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294617    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294620    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294624    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294627    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294629    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294633    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294635    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294640    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294644    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294647    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294650    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294653    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294656    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294660    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294663    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294665    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294668    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294676    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.299298 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294679    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294682    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294691    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294694    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294697    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294700    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294702    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294705    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294707    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294711    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294715    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294717    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294720    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294722    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294725    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294728    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294730    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294733    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294735    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.299945 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294738    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294740    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294743    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294745    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294748    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294750    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294753    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294755    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294758    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294761    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294764    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294767    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294769    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294779    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294782    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294785    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294788    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294791    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294793    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294796    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.300388 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294801    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294804    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294806    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294809    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294811    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294814    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294829    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294832    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294835    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294837    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294840    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294843    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294845    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294848    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294850    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294852    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294855    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294858    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294860    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.300932 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294863    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294866    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294869    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294871    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294874    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294876    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294885    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294888    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294890    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294893    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294896    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294898    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294901    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294904    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294906    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294909    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294911    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294914    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294916    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.301407 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294919    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294921    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294923    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294926    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294928    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294931    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.294955    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.294998    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295003    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295006    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295009    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295011    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295014    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295017    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295020    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295022    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:01.302031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295025    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295028    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295030    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295033    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295045    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295048    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295050    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295053    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295056    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295058    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295061    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295063    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295066    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295068    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295071    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295073    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295076    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295078    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295081    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295083    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:01.302415 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295086    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295089    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295092    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295094    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295097    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295099    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295102    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295105    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295108    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295110    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295113    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295115    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295118    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295120    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295123    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295125    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295128    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295136    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295138    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:01.303007 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295141    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295144    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295146    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295149    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295151    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295154    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295157    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295159    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295162    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295164    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295168    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295172    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295175    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295177    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295181    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295184    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295187    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295189    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295192    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:01.303760 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295194    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295197    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295199    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295204    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295207    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295210    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295212    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295214    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295217    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295220    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295224    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295235    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295238    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295241    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295244    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295246    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295248    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295251    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:01.304552 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295254    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295256    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295261    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295264    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295266    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295269    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295271    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295274    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295276    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295281    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295284    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295287    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295289    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295294    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295297    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295300    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295302    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:01.295306    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:01.305270 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.295372    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.299279    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.300692    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.300796    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.300840    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.303374    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:01.305793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.305182    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:01.315075 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.315057    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:01.320031 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.320014    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:01.321272 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.321231    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:01.324008 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.323986    2579 fs.go:135] Filesystem UUIDs: map[2ce71e8c-355c-4ae8-973d-0b960296f850:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0ae6e77-365f-4524-b2fa-88e0242476f3:/dev/nvme0n1p3]
Oct 30 02:40:01.324085 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.324006    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:40:01.330946 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.330802    2579 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:01.329281074 +0000 UTC m=+0.381911461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099989 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c59bc033f16b27c6a50b002eb406f SystemUUID:ec2c59bc-033f-16b2-7c6a-50b002eb406f BootID:84f01230-c0f3-49bb-85b0-abf9e56c3faa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:bd:11:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:bd:11:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:b5:d8:ba:4c:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:01.330946 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.330937    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:01.331127 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.331022    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:01.333902 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.333870    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:01.334072 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.333903    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:01.334152 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.334076    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:01.334152 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.334089    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:01.334152 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.334130    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:01.334255 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.334154    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:01.335454 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.335440    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:01.340146 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.340131    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:01.340285 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.340274    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:01.341174 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341161    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:01.341228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341180    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:01.341228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341195    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:01.341228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341211    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:01.341228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341223    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:01.341664 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.341651    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:01.350762 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.350743    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:01.357979 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.357963    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:01.358983 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.358969    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:01.359049 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359009    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:01.359195 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359182    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:01.359261 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359202    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:01.359261 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359213    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:01.359261 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359222    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:01.359261 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359230    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:01.359261 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359240    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:01.359849 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359839    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:01.359897 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359852    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:01.359897 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359863    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:01.359897 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359874    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:01.359897 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359887    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:01.360043 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.359902    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:01.360651 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.360641    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:01.360702 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.360655    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:01.364579 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.364564    2579 server.go:1263] "Started kubelet"
Oct 30 02:40:01.364699 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.364646    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:01.364892 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.364756    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:01.365011 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.364923    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:01.365496 ip-10-0-134-164 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:01.366497 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.366480    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:01.366613 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.366599    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:01.371926 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.371907    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:40:01.374256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.374234    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:01.375631 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.375616    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:01.375856 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.375834    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.376402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376380    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:01.376402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376389    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:01.376521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376416    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:01.376521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376515    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:01.376521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376523    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:01.376755 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376735    2579 factory.go:55] Registering systemd factory
Oct 30 02:40:01.376755 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.376755    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:01.377026 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.377011    2579 factory.go:153] Registering CRI-O factory
Oct 30 02:40:01.377026 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.377022    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:01.377150 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.377065    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:01.377150 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.377084    2579 factory.go:103] Registering Raw factory
Oct 30 02:40:01.377150 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.377094    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:01.378288 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.378275    2579 manager.go:319] Starting recovery of all containers
Oct 30 02:40:01.388038 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.388007    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-164.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:40:01.389187 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.388007    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-164.ec2.internal.18e32e50d718300a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-164.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-164.ec2.internal,},FirstTimestamp:2026-10-30 02:40:01.364537354 +0000 UTC m=+0.417167740,LastTimestamp:2026-10-30 02:40:01.364537354 +0000 UTC m=+0.417167740,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-164.ec2.internal,}"
Oct 30 02:40:01.391903 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.391889    2579 manager.go:324] Recovery completed
Oct 30 02:40:01.394382 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.394361    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 02:40:01.397577 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.397564    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.399542 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.399520    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.399599 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.399559    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.399599 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.399573    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.400099 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400078    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:01.400099 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400092    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:01.400187 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400111    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.400187 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400114    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:01.400869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400850    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.400957 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400886    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.400957 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.400903    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.402482 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.402466    2579 policy_none.go:50] "Start"
Oct 30 02:40:01.402544 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.402485    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:01.402544 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.402499    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:01.404592 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.404577    2579 policy_none.go:44] "Start"
Oct 30 02:40:01.405729 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.405706    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:40:01.406026 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.406000    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:40:01.411189 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.411166    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:01.418937 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.418916    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-164.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:40:01.432172 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.432123    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.437988    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-164.ec2.internal.18e32e50d92e54b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-164.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-164.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-164.ec2.internal,},FirstTimestamp:2026-10-30 02:40:01.399542969 +0000 UTC m=+0.452173356,LastTimestamp:2026-10-30 02:40:01.399542969 +0000 UTC m=+0.452173356,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-164.ec2.internal,}"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.438055    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.442458    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlx4z"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.445676    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.445703    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.445712    2579 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.445970    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.445983    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.446133    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.446215    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.446223    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.446812    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.446875    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.451387    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.452694    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.452718    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.452736    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.452767    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:40:01.454148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.453592    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlx4z"
Oct 30 02:40:01.465074 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.464997    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-164.ec2.internal.18e32e50d92eaf4a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-164.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-164.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-164.ec2.internal,},FirstTimestamp:2026-10-30 02:40:01.399566154 +0000 UTC m=+0.452196544,LastTimestamp:2026-10-30 02:40:01.399566154 +0000 UTC m=+0.452196544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-164.ec2.internal,}"
Oct 30 02:40:01.502513 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.502483    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:01.546162 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.546081    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.547741 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.547724    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.547852 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.547755    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.547852 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.547764    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.547852 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.547792    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.552950 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.552925    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"]
Oct 30 02:40:01.552995 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.552990    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.554319 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.554306    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.554367 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.554329    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.554367 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.554338    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.556587 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.556575    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.556731 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.556717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.556783 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.556746    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.558142 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558126    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.558232 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558153    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.558232 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558132    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.558232 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558166    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.558232 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558176    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.558232 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.558186    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.560245 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.560230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.560318 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.560252    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:01.561176 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.561158    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:01.561263 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.561185    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:01.561263 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.561197    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:01.567679 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.567664    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.567744 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.567689    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-164.ec2.internal\": node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.596732 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.596710    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-164.ec2.internal\" not found" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.599259 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.599239    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-164.ec2.internal\" not found" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.610278 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.610257    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.677471 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.677439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.677632 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.677469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.677632 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.677507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.710813 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.710775    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.777781 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.777881 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.777881 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.777982 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777905    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.777982 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777943    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.778048 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.777987    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.811561 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.811503    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:01.902180 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.902152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.904253 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:01.904237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:01.911983 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:01.911869    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.012375 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.012333    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.113200 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.113113    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.213494 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.213449    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.300924 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.300881    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:40:02.301425 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.301052    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:40:02.314197 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.314164    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.414639 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.414606    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.448370 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.448344    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:02.459228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.459183    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:02 +0000 UTC" deadline="2028-06-27 12:36:14.553568902 +0000 UTC"
Oct 30 02:40:02.459228 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.459219    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14553h56m12.094352918s"
Oct 30 02:40:02.494725 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.494693    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:02.495865 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.495842    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:02.495978 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.495880    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:02.495978 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.495898    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:02.509886 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:02.509856    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87383a6fdf9fa44e793f1fdd2d5341d.slice/crio-677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502 WatchSource:0}: Error finding container 677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502: Status 404 returned error can't find the container with id 677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502
Oct 30 02:40:02.511031 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:02.511012    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d17cb448486bb943d1114ac107c4da.slice/crio-7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58 WatchSource:0}: Error finding container 7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58: Status 404 returned error can't find the container with id 7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58
Oct 30 02:40:02.515345 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.515321    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.517515 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.517498    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:02.616441 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.616398    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.655637 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.655575    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:02.716734 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:02.716699    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:02.795992 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.795964    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:02.875968 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.875935    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:02.897808 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.897776    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:02.899103 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.899077    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:02.923019 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:02.922958    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:03.342277 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.342204    2579 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:03.372278 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.372236    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:03.372620 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.372599    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bh7xs","openshift-dns/node-resolver-2gm47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal","kube-system/konnectivity-agent-kgg59","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc","openshift-multus/multus-additional-cni-plugins-86xxs","openshift-ovn-kubernetes/ovnkube-node-jt8jx","openshift-image-registry/node-ca-nrwb4","openshift-multus/multus-2mzsj","openshift-multus/network-metrics-daemon-wdqp2","openshift-network-diagnostics/network-check-target-69lbp","kube-system/global-pull-secret-syncer-j4z9f","openshift-network-operator/iptables-alerter-w27r9"]
Oct 30 02:40:03.375102 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.375077    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:03.375623 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.375602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.378084 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.378064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.378243 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.378129    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:03.380267 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.380242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.382651 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.382631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.385030 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.385011    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.387754 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.387734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.389184 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389272 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389272 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.389348 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389348 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389414 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389349    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389414 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389495 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389495 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq6jm\" (UniqueName: \"kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389495 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389587 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389587 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389646 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.389646 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.389707 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389707 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389770 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389770 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.389830 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.389786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.390693 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.390674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.391649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.391629    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.391649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.391643    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\""
Oct 30 02:40:03.391837 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.391771    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:03.391837 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.391791    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:03.393074 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.393055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.395334 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.395313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.399049 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.398862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.402535 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.402517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.402634 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.402595    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:03.404940 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.404922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:03.405041 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.404982    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:03.443388 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.443197    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:40:03.443388 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.443240    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8kcvv"
Oct 30 02:40:03.454079 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.454056    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8kcvv"
Oct 30 02:40:03.457951 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.457904    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerStarted","Data":"677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502"}
Oct 30 02:40:03.459554 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.459528    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"57d17cb448486bb943d1114ac107c4da","Type":"ContainerStarted","Data":"7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58"}
Oct 30 02:40:03.477385 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.477363    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:03.490565 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490631    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.490696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490706    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdqt\" (UniqueName: \"kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.490928    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:03.490966 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.490962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.491074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:03.991044052 +0000 UTC m=+3.043674455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491221    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.491256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tbf\" (UniqueName: \"kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491410    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491551 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28d9\" (UniqueName: \"kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491790    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.491869 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwct\" (UniqueName: \"kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.491994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492037    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b574r\" (UniqueName: \"kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsft4\" (UniqueName: \"kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492262    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.492465 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w796t\" (UniqueName: \"kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493051 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.492984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6jm\" (UniqueName: \"kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493300    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493326    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493370    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493400    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.493598 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.494105 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.493640    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.494162 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.494144    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.494227 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.494209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.500254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500049    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.500254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500107    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.500254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500147    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:03.500254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500190    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\""
Oct 30 02:40:03.500254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500209    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:03.500697 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500678    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.500817 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500801    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.500817 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500813    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\""
Oct 30 02:40:03.500988 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500868    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:03.500988 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500878    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.500988 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500937    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:03.500988 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500946    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.500989    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501117    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501124    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501120    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501148    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501157    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501167    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\""
Oct 30 02:40:03.501182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501181    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501188    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501219    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501225    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501240    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501351    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501361    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501381    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501382    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501355    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501443    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501430    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\""
Oct 30 02:40:03.501521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.501460    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:03.517434 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.517405    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:03.523029 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.523003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6jm\" (UniqueName: \"kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.540453 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.540429    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:03.593649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwct\" (UniqueName: \"kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b574r\" (UniqueName: \"kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsft4\" (UniqueName: \"kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593829    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.593930 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.593981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w796t\" (UniqueName: \"kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.594366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdqt\" (UniqueName: \"kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594737    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.594927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594779    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tbf\" (UniqueName: \"kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28d9\" (UniqueName: \"kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.594989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595402 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595083    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595418    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595675    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595734    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595771    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595813    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595841    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.595876 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.595867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596166    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596401    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596462    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.596671 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.596622    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.597415 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597386    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.597552 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.597552 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.597552 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.597552 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597546    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597563    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597603    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597608    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.597767 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.598097 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.597942    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:03.598097 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.597981    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.598097 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.598019    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.097989336 +0000 UTC m=+3.150619726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:03.598612 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598584    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.598708 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598621    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.598708 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.598852 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598714    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.598852 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.598967 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598871    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.599020 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.598977    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.599086 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.599028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.599142 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.599079    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.599142 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.599089    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.600892 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.600860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.601097 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.601074    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.601293 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.601214    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.601812 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.601788    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.607447 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.607422    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.620679 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.620655    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:03.620760 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.620682    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:03.620760 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.620696    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:03.620760 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.620759    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.120741614 +0000 UTC m=+3.173372015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:03.623453 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.623421    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsft4\" (UniqueName: \"kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.624211 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.624190    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwct\" (UniqueName: \"kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:03.625463 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.625443    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tbf\" (UniqueName: \"kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.626156 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.626138    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdqt\" (UniqueName: \"kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.626319 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.626285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b574r\" (UniqueName: \"kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.626762 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.626717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w796t\" (UniqueName: \"kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.627778 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.627757    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28d9\" (UniqueName: \"kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.630380 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.630355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.692031 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.691993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:03.700450 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.700423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:03.705660 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.705631    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cd0b32_9830_43e7_a03d_3dceef10e7ac.slice/crio-023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb WatchSource:0}: Error finding container 023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb: Status 404 returned error can't find the container with id 023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb
Oct 30 02:40:03.712412 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.712388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:03.712782 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.712758    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75e3379_3433_4e3f_a6a0_52afc2f47330.slice/crio-57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c WatchSource:0}: Error finding container 57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c: Status 404 returned error can't find the container with id 57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c
Oct 30 02:40:03.720392 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.720373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:03.726840 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.726807    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890281f9_0645_46ac_98d5_93906021aa03.slice/crio-1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190 WatchSource:0}: Error finding container 1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190: Status 404 returned error can't find the container with id 1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190
Oct 30 02:40:03.731161 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.731135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:03.731443 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.731419    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163e0310_6238_4518_9d5a_c297c56f3603.slice/crio-e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46 WatchSource:0}: Error finding container e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46: Status 404 returned error can't find the container with id e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46
Oct 30 02:40:03.735234 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.735215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:03.756192 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.756164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:03.761589 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.761565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:03.776582 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.776554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:03.776879 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.776844    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb7620e_d279_4b5f_863f_a7c5c7820aba.slice/crio-9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e WatchSource:0}: Error finding container 9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e: Status 404 returned error can't find the container with id 9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e
Oct 30 02:40:03.787842 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:03.787799    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8c5bf6_8ec6_4224_a344_4c4da9c21a51.slice/crio-7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f WatchSource:0}: Error finding container 7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f: Status 404 returned error can't find the container with id 7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f
Oct 30 02:40:03.998464 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:03.998391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:03.998599 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.998510    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:03.998599 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:03.998567    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:04.998548422 +0000 UTC m=+4.051178799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:04.098971 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.098939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:04.099136 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.099113    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:04.099204 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.099186    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:05.099165 +0000 UTC m=+4.151795394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:04.199719 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.199680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:04.199965 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.199946    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:04.200065 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.199969    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:04.200065 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.199982    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:04.200065 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:04.200044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:05.200023288 +0000 UTC m=+4.252653687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:04.457795 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.457681    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:04 +0000 UTC" deadline="2028-07-24 19:14:52.339973036 +0000 UTC"
Oct 30 02:40:04.457795 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.457744    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15208h34m47.882232764s"
Oct 30 02:40:04.476315 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.475920    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gm47" event={"ID":"e75e3379-3433-4e3f-a6a0-52afc2f47330","Type":"ContainerStarted","Data":"57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c"}
Oct 30 02:40:04.479252 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.479118    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f"}
Oct 30 02:40:04.482330 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.482244    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerStarted","Data":"9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e"}
Oct 30 02:40:04.484172 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.484144    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrwb4" event={"ID":"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b","Type":"ContainerStarted","Data":"959352dc8af9a86f5ad011d8a7dca62ff010d4f493ec615864ead9b94e211438"}
Oct 30 02:40:04.491126 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.491065    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w27r9" event={"ID":"890281f9-0645-46ac-98d5-93906021aa03","Type":"ContainerStarted","Data":"1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190"}
Oct 30 02:40:04.499080 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.499041    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mzsj" event={"ID":"88cd0b32-9830-43e7-a03d-3dceef10e7ac","Type":"ContainerStarted","Data":"023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb"}
Oct 30 02:40:04.505622 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.504763    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"57d17cb448486bb943d1114ac107c4da","Type":"ContainerStarted","Data":"5dcc551edd9aedcc2235fcb695de77787f08906a648cbe9ce38906aa3eb1fd10"}
Oct 30 02:40:04.515030 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.514932    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh7xs" event={"ID":"533992c7-201e-4aa7-a5eb-0d956cb8cff8","Type":"ContainerStarted","Data":"376eb960ed5c75a754976894ed648b2184686609eb734293b8fc0ef8e3d645be"}
Oct 30 02:40:04.520683 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.520655    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc" event={"ID":"e47ee884-8fa6-49cc-843e-353afd0c281f","Type":"ContainerStarted","Data":"e11bf61e8d76a916e4d27eb2319598ff97d3177162ff6bef90f5dbf28f19194c"}
Oct 30 02:40:04.527766 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.527734    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kgg59" event={"ID":"163e0310-6238-4518-9d5a-c297c56f3603","Type":"ContainerStarted","Data":"e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46"}
Oct 30 02:40:04.546274 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:04.546215    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" podStartSLOduration=1.546196567 podStartE2EDuration="1.546196567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:04.545670399 +0000 UTC m=+3.598300803" watchObservedRunningTime="2026-10-30 02:40:04.546196567 +0000 UTC m=+3.598826969"
Oct 30 02:40:05.008568 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.007908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:05.008568 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.008059    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:05.008568 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.008163    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:07.008141955 +0000 UTC m=+6.060772333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:05.108957 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.108351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:05.108957 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.108522    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:05.108957 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.108587    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:07.108567174 +0000 UTC m=+6.161197553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:05.209235 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.209196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:05.209411 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.209396    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:05.209470 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.209418    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:05.209470 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.209432    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:05.209602 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.209493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:07.20947376 +0000 UTC m=+6.262104158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:05.453389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.453312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:05.453549 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.453457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:05.453912 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.453888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:05.454021 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.453999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:05.454089 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.454081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:05.454193 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:05.454173    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:05.458221 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.458149    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:04 +0000 UTC" deadline="2028-08-03 04:16:23.874212764 +0000 UTC"
Oct 30 02:40:05.458221 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.458201    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15433h36m18.416016294s"
Oct 30 02:40:05.558518 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.558474    2579 generic.go:356] "Generic (PLEG): container finished" podID="b87383a6fdf9fa44e793f1fdd2d5341d" containerID="e983218078749d5f7121c9f26876d574c83e1f107522578c48a700b03a5541b5" exitCode=0
Oct 30 02:40:05.559148 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:05.559122    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerDied","Data":"e983218078749d5f7121c9f26876d574c83e1f107522578c48a700b03a5541b5"}
Oct 30 02:40:06.599586 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:06.599545    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerStarted","Data":"f427d4365a4eeced72ca2f1b02d468205eb7ba3d51808d3d325439aabdc2f79e"}
Oct 30 02:40:06.646484 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:06.646423    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" podStartSLOduration=3.646400079 podStartE2EDuration="3.646400079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:06.64627462 +0000 UTC m=+5.698905020" watchObservedRunningTime="2026-10-30 02:40:06.646400079 +0000 UTC m=+5.699030476"
Oct 30 02:40:07.024230 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.024165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:07.024400 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.024320    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:07.024482 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.024407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:11.024383279 +0000 UTC m=+10.077013670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:07.125292 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.125213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:07.125464 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.125427    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:07.125541 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.125500    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:11.125479967 +0000 UTC m=+10.178110345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:07.227029 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.226289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:07.227029 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.226480    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:07.227029 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.226501    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:07.227029 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.226515    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:07.227029 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.226589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:11.226571025 +0000 UTC m=+10.279201407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:07.461016 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.460168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:07.461016 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.460316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:07.461016 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.460813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:07.461016 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.460931    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:07.461667 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.461520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:07.461667 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:07.461630    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:07.606091 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.605807    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="2b9cc97d58b4ca819bda59dfb3a72d2f3af648e04161c87cd66945c12535a25e" exitCode=0
Oct 30 02:40:07.606804 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:07.606723    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"2b9cc97d58b4ca819bda59dfb3a72d2f3af648e04161c87cd66945c12535a25e"}
Oct 30 02:40:08.622099 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:08.622055    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kgg59" event={"ID":"163e0310-6238-4518-9d5a-c297c56f3603","Type":"ContainerStarted","Data":"76523a300979d680f5bfecac5905f1bfd09af5884727b49f03ffe13ab68df862"}
Oct 30 02:40:09.456126 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:09.456054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:09.456275 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:09.456063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:09.456275 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:09.456203    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:09.456275 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:09.456236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:09.456447 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:09.456336    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:09.456447 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:09.456435    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:09.631327 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:09.631291    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nrwb4" event={"ID":"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b","Type":"ContainerStarted","Data":"869adbdeb405f65c0b96f1755f9da2a0534a128d74547e42654826690fca639c"}
Oct 30 02:40:09.672828 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:09.672775    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kgg59" podStartSLOduration=3.889478999 podStartE2EDuration="7.67275679s" totalImagesPullingTime="3.783277791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:08.671136931 +0000 UTC m=+7.723767329" watchObservedRunningTime="2026-10-30 02:40:09.67275679 +0000 UTC m=+8.725387190"
Oct 30 02:40:10.645662 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:10.645631    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc" event={"ID":"e47ee884-8fa6-49cc-843e-353afd0c281f","Type":"ContainerStarted","Data":"2e95e968243d5bcc28c4cbf0692fe2c5773f2875abaff0b2ec0f8190473d11f2"}
Oct 30 02:40:11.067694 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.066909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:11.067694 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.067105    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:11.067694 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.067207    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.067185513 +0000 UTC m=+18.119815909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:11.168205 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.168170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:11.168396 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.168382    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:11.168467 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.168455    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.168433111 +0000 UTC m=+18.221063509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:11.261343 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.261310    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:11.271923 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.268826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:11.271923 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.269032    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:11.271923 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.269062    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:11.271923 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.269077    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:11.271923 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.269141    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:19.269120654 +0000 UTC m=+18.321751055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:11.456405 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.456246    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:11.261335797Z","UUID":"48527ecf-1eaa-4953-ae98-95d64c79f5d5","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:11.458115 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.458090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:11.458280 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.458251    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:11.459855 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.459568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:11.459855 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.459688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:11.459855 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.459751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:11.459855 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:11.459824    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:11.460815 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.460325    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:11.460815 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.460366    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:11.650756 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:11.650718    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc" event={"ID":"e47ee884-8fa6-49cc-843e-353afd0c281f","Type":"ContainerStarted","Data":"c42b3319ffac1764fcc1b907f6035cdda19de1952d36bff23ddb9322366cc69c"}
Oct 30 02:40:12.253791 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.253756    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:12.255231 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.255206    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:12.295591 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.295483    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nrwb4" podStartSLOduration=4.951528477 podStartE2EDuration="10.29546555s" totalImagesPullingTime="5.343937073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:09.672359406 +0000 UTC m=+8.724989811" watchObservedRunningTime="2026-10-30 02:40:12.29546555 +0000 UTC m=+11.348095946"
Oct 30 02:40:12.655483 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.655307    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2gm47" event={"ID":"e75e3379-3433-4e3f-a6a0-52afc2f47330","Type":"ContainerStarted","Data":"9e7546d47d50061b92da4b11507a93eb41d0423da259519a44e07a3d9e7ca15e"}
Oct 30 02:40:12.658107 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.658080    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="a84cea01f4b9f5a9b4ee28e3b1aa9b2eaa61ead0bf4d37f92d404a8121a96e20" exitCode=0
Oct 30 02:40:12.658906 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.658157    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"a84cea01f4b9f5a9b4ee28e3b1aa9b2eaa61ead0bf4d37f92d404a8121a96e20"}
Oct 30 02:40:12.664213 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.664192    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mzsj" event={"ID":"88cd0b32-9830-43e7-a03d-3dceef10e7ac","Type":"ContainerStarted","Data":"e419c847d64b6c909001c1975e0d6090c6f123df99f5f16efdfcb0874026fb66"}
Oct 30 02:40:12.664290 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.664229    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:12.664630 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.664616    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:12.737717 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.737449    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2gm47" podStartSLOduration=2.164530751 podStartE2EDuration="10.737435031s" totalImagesPullingTime="8.57290428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:12.696835489 +0000 UTC m=+11.749465877" watchObservedRunningTime="2026-10-30 02:40:12.737435031 +0000 UTC m=+11.790065426"
Oct 30 02:40:12.825490 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:12.825389    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2mzsj" podStartSLOduration=2.913482945 podStartE2EDuration="10.825374114s" totalImagesPullingTime="7.911891169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:12.825185761 +0000 UTC m=+11.877816158" watchObservedRunningTime="2026-10-30 02:40:12.825374114 +0000 UTC m=+11.878004511"
Oct 30 02:40:13.453295 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.453268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:13.453998 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:13.453782    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:13.453998 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.453874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:13.453998 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.453903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:13.453998 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:13.453966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:13.454250 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:13.454029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:13.667989 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.667953    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc" event={"ID":"e47ee884-8fa6-49cc-843e-353afd0c281f","Type":"ContainerStarted","Data":"7bc4f296347b917f5d0d3c27df391f005f3005d6d77b6783ce8feed0df44c18e"}
Oct 30 02:40:13.670314 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.670286    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w27r9" event={"ID":"890281f9-0645-46ac-98d5-93906021aa03","Type":"ContainerStarted","Data":"767d21e019fdababc5dee48e472daee37a2016812e7a03de24db26cde3e0022c"}
Oct 30 02:40:13.749463 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:13.749368    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc" podStartSLOduration=3.344737723 podStartE2EDuration="11.749352964s" totalImagesPullingTime="8.404615241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:13.710099254 +0000 UTC m=+12.762729649" watchObservedRunningTime="2026-10-30 02:40:13.749352964 +0000 UTC m=+12.801983362"
Oct 30 02:40:14.676771 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:14.676697    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="d9423d73997b89f13eb2adae0f1a4c6e802ff94c01eb8d2e3b3f00da4b6b9088" exitCode=0
Oct 30 02:40:14.677422 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:14.676791    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"d9423d73997b89f13eb2adae0f1a4c6e802ff94c01eb8d2e3b3f00da4b6b9088"}
Oct 30 02:40:14.678601 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:14.678566    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bh7xs" event={"ID":"533992c7-201e-4aa7-a5eb-0d956cb8cff8","Type":"ContainerStarted","Data":"4fdd93bd0a478614591a94621eb88c80a7cf16a0b4b097d10861c0b64158c550"}
Oct 30 02:40:14.746479 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:14.746427    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w27r9" podStartSLOduration=4.647498971 podStartE2EDuration="12.746413927s" totalImagesPullingTime="8.098914956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:13.749169478 +0000 UTC m=+12.801799869" watchObservedRunningTime="2026-10-30 02:40:14.746413927 +0000 UTC m=+13.799044370"
Oct 30 02:40:14.788117 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:14.788065    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bh7xs" podStartSLOduration=2.232812014 podStartE2EDuration="12.788048117s" totalImagesPullingTime="10.555236103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:14.7878045 +0000 UTC m=+13.840434895" watchObservedRunningTime="2026-10-30 02:40:14.788048117 +0000 UTC m=+13.840678510"
Oct 30 02:40:15.453928 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:15.453894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:15.454080 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:15.453936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:15.454080 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:15.453894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:15.454080 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:15.454016    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:15.454215 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:15.454092    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:15.454260 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:15.454199    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:16.686957 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:16.686729    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="b87abfbb6933a9656bf559d640f7e2fc1d44ffcdd6091c95b3cc5269c3ad0209" exitCode=0
Oct 30 02:40:16.686957 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:16.686794    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"b87abfbb6933a9656bf559d640f7e2fc1d44ffcdd6091c95b3cc5269c3ad0209"}
Oct 30 02:40:17.454036 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:17.453833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:17.454036 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:17.453833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:17.454036 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:17.453984    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:17.454036 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:17.453836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:17.454355 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:17.454008    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:17.454355 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:17.454105    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:19.134792 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.134582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:19.135203 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.134733    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:19.135203 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.134906    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret podName:d80631ef-63c9-400c-a46f-6eed3ca27590 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.134883145 +0000 UTC m=+34.187513542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret") pod "global-pull-secret-syncer-j4z9f" (UID: "d80631ef-63c9-400c-a46f-6eed3ca27590") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:40:19.235880 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.235841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:19.236040 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.235980    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:19.236040 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.236037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.236024744 +0000 UTC m=+34.288655118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:40:19.336416 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.336381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:19.336560 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.336520    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:40:19.336560 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.336533    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:40:19.336560 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.336543    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bwct5 for pod openshift-network-diagnostics/network-check-target-69lbp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:19.336656 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.336589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5 podName:d61b3b7e-953d-48e5-b08d-792f90f55ce8 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.336577041 +0000 UTC m=+34.389207416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwct5" (UniqueName: "kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5") pod "network-check-target-69lbp" (UID: "d61b3b7e-953d-48e5-b08d-792f90f55ce8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:40:19.454027 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.453941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:19.454176 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.454099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:19.454176 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.454118    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:19.454274 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.454169    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:19.454274 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.454203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:19.454274 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:19.454267    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:19.695406 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695371    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"af8686c7f44e393b7b973a299ad1231906824be992d8d33f93ff774be03bf59c"}
Oct 30 02:40:19.695562 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695443    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"c165dea971506f046c2865bb3bd6b9fe7ea235b72ee2862674ef7905f9800873"}
Oct 30 02:40:19.695562 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695475    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"9c861f12d0d0ac18ca6245dd23441f74a37e79e1a3f8de8373e396faa978e900"}
Oct 30 02:40:19.695562 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695512    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"f2f907ce9ffdbef573dda2fcaa4decf71af50f521bbade252a166e8e86dcaa17"}
Oct 30 02:40:19.695562 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695545    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"8fbd8381013f278c7d9b8b1d920ffc51a5a48a8919e2ada60dd68c073323b9f3"}
Oct 30 02:40:19.695720 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.695580    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"c297f24cb7d43163e379ba0938167af3202d38d44047149d268be7a9bc161a4d"}
Oct 30 02:40:19.697945 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.697920    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="81fb1f792789083bf3889f8fb047b9ea4c3026e1b1e1501fa42bc3f4ffb43275" exitCode=0
Oct 30 02:40:19.698047 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:19.697958    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"81fb1f792789083bf3889f8fb047b9ea4c3026e1b1e1501fa42bc3f4ffb43275"}
Oct 30 02:40:20.702650 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:20.702617    2579 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="806fe11cd6bf43b5c870612556231af0ec84c9b4720c65063415f66df24d8ebd" exitCode=0
Oct 30 02:40:20.703114 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:20.702675    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerDied","Data":"806fe11cd6bf43b5c870612556231af0ec84c9b4720c65063415f66df24d8ebd"}
Oct 30 02:40:21.453777 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.453698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:21.453951 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:21.453794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:21.453951 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.453853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:21.453951 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.453885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:21.454095 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:21.453953    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:21.454095 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:21.454033    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:21.707916 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.707811    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"78f0c1b5dbbfb54aa526661f3a800134b0de6946cfa520d2a1459adaa07ba05f"}
Oct 30 02:40:21.710769 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.710742    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86xxs" event={"ID":"adb7620e-d279-4b5f-863f-a7c5c7820aba","Type":"ContainerStarted","Data":"460e431278662f9d0c922225387c600ecf00c6c2845b268b17ab190f0e543991"}
Oct 30 02:40:21.780732 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:21.780683    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86xxs" podStartSLOduration=8.339843999 podStartE2EDuration="19.780670138s" totalImagesPullingTime="11.440826139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:21.76873865 +0000 UTC m=+20.821369046" watchObservedRunningTime="2026-10-30 02:40:21.780670138 +0000 UTC m=+20.833300534"
Oct 30 02:40:23.453456 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.453264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:23.454094 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.453271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:23.454094 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:23.453576    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:23.454094 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.453298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:23.454094 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:23.453682    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:23.454094 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:23.453608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:23.718405 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.718295    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" event={"ID":"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51","Type":"ContainerStarted","Data":"6db5756f48b112aa5224e1dda309a4c129eeca46db17b7acde72b9b38acd515a"}
Oct 30 02:40:23.718580 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.718545    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:23.718649 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.718634    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:23.733371 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.733347    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:23.733506 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.733459    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:23.805833 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:23.805773    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx" podStartSLOduration=6.7849905029999995 podStartE2EDuration="21.805759012s" totalImagesPullingTime="15.020768509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:23.805730717 +0000 UTC m=+22.858361112" watchObservedRunningTime="2026-10-30 02:40:23.805759012 +0000 UTC m=+22.858389408"
Oct 30 02:40:24.720814 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:24.720786    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:25.336673 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.336605    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69lbp"]
Oct 30 02:40:25.336854 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.336740    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:25.336854 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:25.336829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:25.339256 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.339231    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4z9f"]
Oct 30 02:40:25.339389 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.339322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:25.339461 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:25.339402    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:25.348412 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.348390    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdqp2"]
Oct 30 02:40:25.348516 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.348480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:25.348564 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:25.348549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:25.723236 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:25.723164    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:27.453070 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:27.453037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:27.453458 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:27.453144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:27.453458 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:27.453156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:27.453458 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:27.453145    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:27.453458 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:27.453238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:27.453458 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:27.453299    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:28.912016 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:28.911983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:29.094521 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:29.094488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:29.306854 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:29.306749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:29.453367 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:29.453334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:29.453545 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:29.453400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:29.453545 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:29.453400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:29.453545 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:29.453507    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdqp2" podUID="8c88dc4f-1666-4eec-a7cc-17b63c391400"
Oct 30 02:40:29.453687 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:29.453582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4z9f" podUID="d80631ef-63c9-400c-a46f-6eed3ca27590"
Oct 30 02:40:29.453687 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:29.453632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-69lbp" podUID="d61b3b7e-953d-48e5-b08d-792f90f55ce8"
Oct 30 02:40:30.262880 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:30.262786    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:30.263239 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:30.263005    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:30.273776 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:30.273615    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:31.269247 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.269178    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeReady"
Oct 30 02:40:31.269634 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.269293    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:40:31.333080 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.333053    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cr8"]
Oct 30 02:40:31.340801 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.340774    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7kjhk"]
Oct 30 02:40:31.340941 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.340897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.343710 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.343686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.357591 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.357571    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cr8"]
Oct 30 02:40:31.360091 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.360065    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:40:31.360217 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.360202    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:40:31.360278 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.360268    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\""
Oct 30 02:40:31.360796 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.360776    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7kjhk"]
Oct 30 02:40:31.368971 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.368950    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:40:31.368971 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.368950    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:40:31.371647 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.371630    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\""
Oct 30 02:40:31.371705 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.371630    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:40:31.417463 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.417463 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586wm\" (UniqueName: \"kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.417684 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.417684 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.417684 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttn4g\" (UniqueName: \"kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.417684 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.417657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.456331 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.456305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:31.456331 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.456316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:31.456331 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.456330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:31.471635 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.471608    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:31.471635 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.471624    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\""
Oct 30 02:40:31.471635 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.471617    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:31.471635 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.471626    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\""
Oct 30 02:40:31.473791 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.473771    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\""
Oct 30 02:40:31.474092 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.474066    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:31.474336 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.474313    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:31.519061 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.519241 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttn4g\" (UniqueName: \"kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.519241 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.519241 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.519241 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586wm\" (UniqueName: \"kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.519241 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.519514 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.519514 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:31.519381    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:31.519514 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:31.519440    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:32.019421043 +0000 UTC m=+31.072051430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:31.519727 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.519710    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.523128 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.523106    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.544077 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.544038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttn4g\" (UniqueName: \"kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:31.544317 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.544294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586wm\" (UniqueName: \"kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.640333 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.640293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:31.653706 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.653676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:31.763582 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.763549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:31.914206 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.914172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:31.970210 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:31.970174    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cr8"]
Oct 30 02:40:31.974832 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:31.974790    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a135dd3_ea50_4c74_86d7_38c5e2d36ca9.slice/crio-065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181 WatchSource:0}: Error finding container 065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181: Status 404 returned error can't find the container with id 065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181
Oct 30 02:40:32.022903 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:32.022874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:32.023060 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:32.022994    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:32.023060 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:32.023044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:33.023031236 +0000 UTC m=+32.075661610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:32.457747 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:32.457712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:32.594560 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:32.594510    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:32.738552 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:32.738473    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2cr8" event={"ID":"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9","Type":"ContainerStarted","Data":"065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181"}
Oct 30 02:40:33.032299 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.032216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:33.032454 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.032393    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:33.032529 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.032471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.032450041 +0000 UTC m=+34.085080421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:33.381413 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.381376    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2hqz8"]
Oct 30 02:40:33.395320 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.395293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.406505 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.406479    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hqz8"]
Oct 30 02:40:33.451948 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.451919    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-j7nq4-qbr4g"]
Oct 30 02:40:33.468903 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.468868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.475430 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.475403    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j7nq4-qbr4g"]
Oct 30 02:40:33.479407 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.479378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:33.491047 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.490924    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:40:33.491047 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.490936    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:40:33.491047 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.490936    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:40:33.491047 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.490967    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:40:33.491408 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.491125    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:40:33.491408 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.491207    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpstg\""
Oct 30 02:40:33.491408 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.491251    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\""
Oct 30 02:40:33.507647 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.507624    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:40:33.537195 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.537160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.537329 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.537224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.537329 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.537305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.537392 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.537337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhqw\" (UniqueName: \"kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.537450 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.537431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638097 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.637999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.638097 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638300 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhqw\" (UniqueName: \"kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638300 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.638300 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.638204    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:33.638300 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.638286    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.138266767 +0000 UTC m=+33.190897167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:33.638457 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638457 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.638457 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.638457 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638633 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638665 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.638652    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:33.638665 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638763 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.638684    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.638763 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:33.638698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:34.138683642 +0000 UTC m=+33.191314017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:33.662375 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.662342    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhqw\" (UniqueName: \"kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:33.669793 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.669762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:33.686660 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.686628    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"]
Oct 30 02:40:33.719562 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.719534    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"]
Oct 30 02:40:33.719706 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.719652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:33.734095 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.734068    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:40:33.734613 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.734595    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fd952\""
Oct 30 02:40:33.739079 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.739209 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.739267 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.739267 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.739737 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.739737 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.739726    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.740203 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.740181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.765770 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.765740    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.797660 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.797601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:33.805382 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.805341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:33.834888 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.834863    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:33.840208 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.840178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8v4\" (UniqueName: \"kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4\") pod \"migrator-66df5d496b-t6hs5\" (UID: \"c745d5c2-e685-49c5-90f5-9b6b0946acca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:33.941609 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.941521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8v4\" (UniqueName: \"kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4\") pod \"migrator-66df5d496b-t6hs5\" (UID: \"c745d5c2-e685-49c5-90f5-9b6b0946acca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:33.943194 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.943162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:33.965836 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:33.965797    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8v4\" (UniqueName: \"kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4\") pod \"migrator-66df5d496b-t6hs5\" (UID: \"c745d5c2-e685-49c5-90f5-9b6b0946acca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:34.041988 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.041953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:34.119441 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.119384    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j7nq4-qbr4g"]
Oct 30 02:40:34.122901 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:34.122873    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c945bb_127b_4ce1_9807_dc3de3cb297b.slice/crio-58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c WatchSource:0}: Error finding container 58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c: Status 404 returned error can't find the container with id 58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.143597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.143715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:34.143897    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:34.143992    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.143957704 +0000 UTC m=+34.196588092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:34.144339    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:34.146713 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:34.144533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:35.144462266 +0000 UTC m=+34.197092655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:34.249113 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.249014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:34.395345 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.395306    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"]
Oct 30 02:40:34.399537 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:34.399510    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc745d5c2_e685_49c5_90f5_9b6b0946acca.slice/crio-2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3 WatchSource:0}: Error finding container 2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3: Status 404 returned error can't find the container with id 2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3
Oct 30 02:40:34.413824 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.413795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:34.547589 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.547517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:34.744997 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.744966    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5" event={"ID":"c745d5c2-e685-49c5-90f5-9b6b0946acca","Type":"ContainerStarted","Data":"2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3"}
Oct 30 02:40:34.746252 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.746223    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g" event={"ID":"43c945bb-127b-4ce1-9807-dc3de3cb297b","Type":"ContainerStarted","Data":"58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c"}
Oct 30 02:40:34.780254 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.780229    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2m9bp"]
Oct 30 02:40:34.797464 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.797439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:34.806200 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.806175    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2m9bp"]
Oct 30 02:40:34.817623 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.817593    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:40:34.817623 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.817619    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:40:34.817841 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.817654    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:40:34.817906 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.817870    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:40:34.817961 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.817931    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rv7wz\""
Oct 30 02:40:34.950189 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.950148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:34.950366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.950211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56kd\" (UniqueName: \"kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:34.950366 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:34.950253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.051425 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.051393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.051576 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.051450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56kd\" (UniqueName: \"kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.051576 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.051487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:35.051576 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.051513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.052263 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.051854    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:35.052263 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.051919    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:39.05189791 +0000 UTC m=+38.104528308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:35.052446 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.052319    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.054548 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.054521    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.084646 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.084579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56kd\" (UniqueName: \"kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.131182 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.130977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:35.152714 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.152671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:35.152920 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.152780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:35.152920 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.152829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:35.153963 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.153397    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:35.153963 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.153474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:37.153454121 +0000 UTC m=+36.206084499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:35.153963 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.153878    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:35.153963 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.153931    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:37.153915506 +0000 UTC m=+36.206545894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:35.156114 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.156087    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:35.168696 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.168659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:35.253538 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.253502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:35.253703 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.253677    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:35.253785 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:35.253756    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:41:07.253735651 +0000 UTC m=+66.306366031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : secret "metrics-daemon-secret" not found
Oct 30 02:40:35.329594 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.329548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:35.354788 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.354713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:35.357525 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.357498    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:35.403498 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.403466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:35.419605 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.419577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:35.463672 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.463641    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2m9bp"]
Oct 30 02:40:35.471846 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:35.470974    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1292202_817e_4114_b11f_91f3e6c65f58.slice/crio-ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96 WatchSource:0}: Error finding container ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96: Status 404 returned error can't find the container with id ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96
Oct 30 02:40:35.738116 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.738084    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-69lbp"]
Oct 30 02:40:35.743275 ip-10-0-134-164 kubenswrapper[2579]: W1030 02:40:35.743245    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61b3b7e_953d_48e5_b08d_792f90f55ce8.slice/crio-4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff WatchSource:0}: Error finding container 4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff: Status 404 returned error can't find the container with id 4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff
Oct 30 02:40:35.749927 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.749895    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2cr8" event={"ID":"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9","Type":"ContainerStarted","Data":"7dd6e62dc6210692652cb3b6f00488a6d0112147899c91b5162acac4e1f4d3c4"}
Oct 30 02:40:35.751183 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.751155    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp" event={"ID":"c1292202-817e-4114-b11f-91f3e6c65f58","Type":"ContainerStarted","Data":"ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96"}
Oct 30 02:40:35.752326 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.752303    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69lbp" event={"ID":"d61b3b7e-953d-48e5-b08d-792f90f55ce8","Type":"ContainerStarted","Data":"4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff"}
Oct 30 02:40:35.770561 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.770539    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4z9f"]
Oct 30 02:40:35.793974 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:35.793936    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b2cr8" podStartSLOduration=2.033544605 podStartE2EDuration="4.793919994s" totalImagesPullingTime="2.760375389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:35.792872159 +0000 UTC m=+34.845502557" watchObservedRunningTime="2026-10-30 02:40:35.793919994 +0000 UTC m=+34.846550392"
Oct 30 02:40:36.130961 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:36.130934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:36.757013 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:36.756979    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4z9f" event={"ID":"d80631ef-63c9-400c-a46f-6eed3ca27590","Type":"ContainerStarted","Data":"21696c652dd24198f57a224e1276935f65b7ddbdb4307c4d0cf4d4fffa4a5798"}
Oct 30 02:40:36.759444 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:36.759214    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5" event={"ID":"c745d5c2-e685-49c5-90f5-9b6b0946acca","Type":"ContainerStarted","Data":"76353224576d8b45c938fd627155848b403e2b4ee4400ea9e73103ed931d3380"}
Oct 30 02:40:36.763612 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:36.763555    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g" event={"ID":"43c945bb-127b-4ce1-9807-dc3de3cb297b","Type":"ContainerStarted","Data":"5229218a2781a60c2ff1a647cadf1d85cffdeabfe968371bd9e8870f6398b0f7"}
Oct 30 02:40:36.806010 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:36.804890    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5" podStartSLOduration=2.319525862 podStartE2EDuration="3.804871272s" totalImagesPullingTime="1.48534541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:36.804033273 +0000 UTC m=+35.856663663" watchObservedRunningTime="2026-10-30 02:40:36.804871272 +0000 UTC m=+35.857501671"
Oct 30 02:40:37.170676 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:37.170638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:37.171177 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:37.170814    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:37.171177 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:37.170994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:37.171177 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:37.171073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.171047286 +0000 UTC m=+40.223677660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:37.171177 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:37.171105    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:40:37.171177 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:37.171155    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:41.171140428 +0000 UTC m=+40.223770815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : configmap "kube-rbac-proxy" not found
Oct 30 02:40:38.769711 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:38.769635    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp" event={"ID":"c1292202-817e-4114-b11f-91f3e6c65f58","Type":"ContainerStarted","Data":"8cf52294a5f00e3a24b346eb5b2f1d00762ecfb5bad9fcae3a70b143766e7615"}
Oct 30 02:40:38.809666 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:38.809608    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g" podStartSLOduration=3.39668244 podStartE2EDuration="5.80959072s" totalImagesPullingTime="2.41290828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:36.846597443 +0000 UTC m=+35.899227840" watchObservedRunningTime="2026-10-30 02:40:38.80959072 +0000 UTC m=+37.862221118"
Oct 30 02:40:38.811217 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:38.810267    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp" podStartSLOduration=1.867425871 podStartE2EDuration="4.810258563s" totalImagesPullingTime="2.942832692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:38.809118179 +0000 UTC m=+37.861748580" watchObservedRunningTime="2026-10-30 02:40:38.810258563 +0000 UTC m=+37.862888960"
Oct 30 02:40:39.086293 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:39.086221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:39.086427 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:39.086351    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:39.086427 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:39.086411    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:47.086397105 +0000 UTC m=+46.139027495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:39.362479 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:39.362390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:39.565829 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:39.565792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:39.791667 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:39.791630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:39.969597 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:39.969555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:40.079165 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.079101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gm47_e75e3379-3433-4e3f-a6a0-52afc2f47330/dns-node-resolver/0.log"
Oct 30 02:40:40.208083 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.208057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:40.540777 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.540695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:40.776535 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.776496    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-69lbp" event={"ID":"d61b3b7e-953d-48e5-b08d-792f90f55ce8","Type":"ContainerStarted","Data":"ceaa9329287f61759d5171c52b231ae885e25deaa5fa950c4fc6100a4db7caf9"}
Oct 30 02:40:40.776789 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.776760    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:40.854187 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.854032    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-69lbp" podStartSLOduration=34.549284102 podStartE2EDuration="38.85401657s" totalImagesPullingTime="4.304732468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:40.852766619 +0000 UTC m=+39.905397017" watchObservedRunningTime="2026-10-30 02:40:40.85401657 +0000 UTC m=+39.906646965"
Oct 30 02:40:40.930492 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:40.930463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:41.071724 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.071699    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrwb4_5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b/node-ca/0.log"
Oct 30 02:40:41.204410 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.204373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:41.204564 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.204429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:41.204564 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:41.204533    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:41.204647 ip-10-0-134-164 kubenswrapper[2579]: E1030 02:40:41.204601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:49.204584644 +0000 UTC m=+48.257215023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:41.204881 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.204865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:41.327735 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.327698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:41.500129 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.500099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:41.631325 ip-10-0-134-164 kubenswrapper[2579]: I1030 02:40:41.631284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:41.703034 ip-10-0-134-164 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:40:41.710427 ip-10-0-134-164 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:40:41.710587 ip-10-0-134-164 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:40:41.710634 ip-10-0-134-164 systemd[1]: kubelet.service: Consumed 2.559s CPU time.
Oct 30 02:40:41.721376 ip-10-0-134-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:40:41.813013 ip-10-0-134-164 kubenswrapper[4528]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:41.813013 ip-10-0-134-164 kubenswrapper[4528]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:40:41.813013 ip-10-0-134-164 kubenswrapper[4528]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:41.813013 ip-10-0-134-164 kubenswrapper[4528]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:40:41.815956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815865    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.815956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815953    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.815956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815957    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.815956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815960    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.815956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815963    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815966    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815969    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815972    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815974    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815977    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815980    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815982    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815985    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815987    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815991    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815994    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.815997    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816000    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816002    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816005    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816007    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816010    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816012    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.816153 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816015    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816019    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816023    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816027    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816030    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816033    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816036    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816038    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816045    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816048    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816050    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816054    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816057    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816059    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816062    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816066    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816068    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816071    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816074    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816077    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.816628 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816079    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816082    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816084    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816087    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816090    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816092    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816095    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816098    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816100    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816103    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816105    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816108    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816111    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816113    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816116    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816118    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816127    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816130    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816133    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816136    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.817132 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816139    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816142    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816144    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816147    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816150    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816154    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816159    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816162    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816165    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816168    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816171    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816173    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816176    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816178    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816181    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816184    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816187    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816189    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816192    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.817684 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816194    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816204    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816207    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816210    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816213    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816215    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816218    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816221    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816223    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816227    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816229    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816232    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816235    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816238    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816240    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816243    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816245    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816249    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816252    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816254    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.818171 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816258    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816261    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816263    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.816266    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817126    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817135    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817139    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817142    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817146    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817149    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817151    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817154    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817157    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817160    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817165    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817170    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817174    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817179    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817183    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.818724 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817186    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817189    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817192    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817195    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817198    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817201    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817204    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817206    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817209    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817211    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817215    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817218    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817221    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817223    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817226    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817229    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817231    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817234    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817236    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817239    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.819419 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817241    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817244    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817247    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817252    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817256    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817260    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817264    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817269    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817272    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817275    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817280    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817284    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817287    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817290    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817293    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817296    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817298    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817301    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.819936 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817303    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817306    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817309    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817312    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817314    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817317    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817320    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817322    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817326    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817328    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817332    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817336    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817340    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817345    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817349    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817353    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817358    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817361    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817364    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817367    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.820440 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817370    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817372    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817375    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817379    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817384    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817388    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817391    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817395    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817398    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817400    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817403    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817406    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817408    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817411    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817413    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817416    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817419    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817421    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817426    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.820977 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817430    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817435    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817439    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817445    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817448    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817451    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817453    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817456    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817459    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817462    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817467    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817471    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817474    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817476    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817983    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817992    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817996    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.817999    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818002    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818005    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.821454 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818008    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818010    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818013    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818016    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818019    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818021    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818024    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818026    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818029    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818032    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818034    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818037    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818042    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818044    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818047    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818049    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818052    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818055    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818057    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818060    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.821978 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818062    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818065    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818068    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818070    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818073    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818076    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818078    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818083    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818086    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818089    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818093    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818097    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818100    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818102    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818105    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818108    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818111    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818114    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818117    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.822480 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818119    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818122    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818125    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818127    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818130    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818134    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818137    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818139    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818142    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818145    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818147    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818150    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818152    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818155    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818158    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818161    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818163    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818166    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818169    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.822951 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818171    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818176    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818179    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818182    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818185    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818188    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818190    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818193    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818196    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818198    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818201    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818204    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818206    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818209    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818211    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818214    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818217    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818219    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818222    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.823543 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818226    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818230    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818234    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818238    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818240    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818243    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818245    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818248    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818251    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818253    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818256    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818258    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818261    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818263    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818268    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818270    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818273    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818275    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818279    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818281    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.824037 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818284    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818287    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.818290    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818391    4528 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818410    4528 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818419    4528 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818424    4528 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818428    4528 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818432    4528 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818437    4528 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818446    4528 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818450    4528 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818453    4528 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818456    4528 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818460    4528 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818463    4528 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818466    4528 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818469    4528 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818472    4528 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818475    4528 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818478    4528 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818482    4528 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818485    4528 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818488    4528 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:40:41.824531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818492    4528 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818496    4528 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818506    4528 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818509    4528 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818512    4528 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818516    4528 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818519    4528 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818522    4528 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818525    4528 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818528    4528 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818531    4528 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818536    4528 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818539    4528 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818541    4528 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818544    4528 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818547    4528 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818550    4528 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818555    4528 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818559    4528 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818562    4528 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818565    4528 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818568    4528 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818572    4528 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818575    4528 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818578    4528 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:40:41.825214 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818581    4528 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818584    4528 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818587    4528 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818590    4528 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818594    4528 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818597    4528 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818599    4528 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818602    4528 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818606    4528 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818611    4528 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818615    4528 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818618    4528 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818621    4528 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818624    4528 flags.go:64] FLAG: --help="false"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818627    4528 flags.go:64] FLAG: --hostname-override="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818630    4528 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818633    4528 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818636    4528 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818640    4528 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818643    4528 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818646    4528 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818649    4528 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818652    4528 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:40:41.825905 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818655    4528 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818658    4528 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818661    4528 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818664    4528 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818666    4528 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818669    4528 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818672    4528 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818675    4528 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818678    4528 flags.go:64] FLAG: --lock-file=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818681    4528 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818684    4528 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818687    4528 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818693    4528 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818698    4528 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818701    4528 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818704    4528 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818707    4528 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818710    4528 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818716    4528 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818719    4528 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818724    4528 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818728    4528 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818732    4528 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818735    4528 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818738    4528 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:40:41.826476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818742    4528 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818745    4528 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818748    4528 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818751    4528 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818754    4528 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818761    4528 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818765    4528 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818768    4528 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818771    4528 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818774    4528 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818777    4528 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818780    4528 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818783    4528 flags.go:64] FLAG: --port="10250"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818786    4528 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818789    4528 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0451b8a459baa5f0b"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818792    4528 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818796    4528 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818799    4528 flags.go:64] FLAG: --register-node="true"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818802    4528 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818805    4528 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818813    4528 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818826    4528 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818830    4528 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818834    4528 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818837    4528 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818841    4528 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:40:41.827132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818844    4528 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818847    4528 flags.go:64] FLAG: --runonce="false"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818850    4528 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818853    4528 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818856    4528 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818858    4528 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818861    4528 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818870    4528 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818874    4528 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818877    4528 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818880    4528 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818883    4528 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818886    4528 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818889    4528 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818892    4528 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818895    4528 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818898    4528 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818904    4528 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818906    4528 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818909    4528 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818913    4528 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818916    4528 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818919    4528 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818922    4528 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818925    4528 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:40:41.827769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818928    4528 flags.go:64] FLAG: --v="2"
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818934    4528 flags.go:64] FLAG: --version="false"
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818938    4528 flags.go:64] FLAG: --vmodule=""
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818942    4528 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.818946    4528 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819070    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819073    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819077    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819079    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819082    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819084    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819088    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819090    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819093    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819095    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819099    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819102    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819104    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819107    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819110    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.828377 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819112    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819115    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819118    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819120    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819123    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819126    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819129    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819131    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819134    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819136    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819139    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819142    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819145    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819149    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819152    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819155    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819157    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819160    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819163    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.828931 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819166    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819168    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819171    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819173    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819176    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819179    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819181    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819184    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819186    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819190    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819192    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819195    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819198    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819200    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819203    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819212    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819216    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819218    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819221    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819224    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.829561 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819226    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819229    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819231    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819235    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819239    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819243    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819247    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819251    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819254    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819256    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819259    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819261    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819264    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819267    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819270    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819274    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819277    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819280    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819283    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.830361 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819287    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819290    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819292    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819296    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819298    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819301    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819303    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819306    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819309    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819312    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819315    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819317    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819320    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819323    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819325    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819328    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819331    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819333    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819336    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819340    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.830973 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819343    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819345    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819348    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819351    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819353    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819356    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819358    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819361    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819363    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819366    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819368    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819371    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819374    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.819400    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819434    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819439    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.831559 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819442    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819445    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819447    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819450    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819453    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819455    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819461    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819463    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819466    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819469    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819471    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819474    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819477    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819479    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819482    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819489    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819492    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819495    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819497    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819500    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.832053 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819502    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819505    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819508    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819511    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819513    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819516    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819518    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819521    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819523    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819526    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819529    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819531    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819534    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819537    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819540    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819543    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819545    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819548    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819552    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.832623 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819554    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819557    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819559    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819562    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819565    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819568    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819571    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819574    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819577    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819580    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819583    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819585    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819588    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819591    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819593    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819596    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819598    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819602    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819605    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.833269 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819607    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819610    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819612    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819615    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819617    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819620    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819622    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819625    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819628    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819631    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819633    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819636    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819640    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819643    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819645    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819648    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819651    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819653    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819656    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819658    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.833883 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819661    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819665    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819667    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819672    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819675    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819679    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819682    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819685    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819688    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819691    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819693    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819696    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819698    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819701    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819705    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819708    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819711    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819714    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819717    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.834380 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819719    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819722    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819725    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819727    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819730    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819734    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.819736    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.824240    4528 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.824257    4528 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.824274    4528 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.824279    4528 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824345    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824350    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824353    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824356    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824358    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824361    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824363    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824366    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824369    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.834856 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824371    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824374    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824377    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824379    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824383    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824386    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824389    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824391    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824394    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824396    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824399    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824401    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824404    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824407    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824409    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824412    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824414    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824418    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824420    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824423    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.835383 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824427    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824430    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824433    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824436    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824438    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824441    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824444    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824446    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824449    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824452    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824454    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824457    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824459    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824462    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824464    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824467    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824470    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824472    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824475    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824478    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.836114 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824481    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824483    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824486    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824489    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824491    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824494    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824496    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824499    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824502    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824504    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824507    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824509    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824512    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824519    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824522    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824525    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824527    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824530    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824533    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.836768 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824535    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824538    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824541    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824544    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824546    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824549    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824552    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824554    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824557    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824560    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824562    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824566    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824568    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824571    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824573    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824576    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824579    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824581    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824584    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.837353 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824586    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824589    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824591    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824594    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824597    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824599    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824602    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824607    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824611    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824615    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824618    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824620    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824623    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824627    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824631    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824634    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824637    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824639    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.837851 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824642    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.824669    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824831    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824838    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824843    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824846    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824849    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824852    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824854    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824857    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824860    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824863    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824865    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824868    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824871    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824873    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.838285 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824876    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824878    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824881    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824884    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824887    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824891    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824895    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824899    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824902    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824905    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824908    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824910    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824913    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824916    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824918    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824921    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824924    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824927    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824929    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824932    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.838675 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824934    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824937    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824939    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824944    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824947    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824950    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824952    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824955    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824958    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824961    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824964    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824966    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824969    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824971    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824974    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824980    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824982    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824985    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824988    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.839312 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824990    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824994    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.824997    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825000    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825002    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825005    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825007    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825010    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825012    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825015    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825018    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825021    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825023    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825026    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825028    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825031    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825034    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825036    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825039    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825042    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.839770 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825044    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825047    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825050    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825053    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825055    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825058    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825060    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825063    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825069    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825071    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825074    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825077    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825080    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825082    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825085    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825088    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825090    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825093    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825095    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825098    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.840255 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825100    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825103    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825105    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825108    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825111    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825113    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825116    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825118    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825121    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825124    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825126    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825129    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825131    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.825156    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825185    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825189    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:40:41.840763 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825192    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825195    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825198    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825201    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825203    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825207    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825210    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825212    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825215    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825218    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825221    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825224    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825227    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825229    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825232    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825235    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825237    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825240    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825243    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825245    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:40:41.841166 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825248    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825250    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825253    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825256    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825259    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825261    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825264    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825266    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825269    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825271    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825274    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825277    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825279    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825282    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825284    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825287    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825289    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825293    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825297    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:40:41.841645 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825301    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825303    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825306    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825309    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825311    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825314    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825317    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825320    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825322    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825325    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825328    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825330    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825333    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825335    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825338    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825341    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825343    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825349    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825353    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:40:41.842127 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825356    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825359    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825362    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825365    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825367    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825370    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825372    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825375    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825377    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825380    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825382    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825386    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825389    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825392    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825394    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825397    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825400    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825403    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825405    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:40:41.842601 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825408    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825411    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825413    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825416    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825418    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825421    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825424    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825426    4528 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825429    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825432    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825434    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825438    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825441    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825444    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825447    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825449    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825452    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825454    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825457    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825459    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:40:41.843327 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825462    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825465    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825467    4528 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825470    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825475    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825478    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:41.825480    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.825529    4528 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826448    4528 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826501    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826679    4528 server.go:1014] "Starting client certificate rotation"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826829    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826875    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:35:02 +0000 UTC" deadline="2028-03-31 12:05:10.380636081 +0000 UTC"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.826914    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12441h24m28.553725338s"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.827122    4528 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.828238    4528 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:40:41.844165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.830668    4528 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:40:41.844859 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.832363    4528 log.go:25] "Validated CRI v1 image API"
Oct 30 02:40:41.844859 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.833159    4528 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:40:41.844859 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.834697    4528 fs.go:135] Filesystem UUIDs: map[2ce71e8c-355c-4ae8-973d-0b960296f850:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a0ae6e77-365f-4524-b2fa-88e0242476f3:/dev/nvme0n1p3]
Oct 30 02:40:41.845007 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.834719    4528 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21696c652dd24198f57a224e1276935f65b7ddbdb4307c4d0cf4d4fffa4a5798/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21696c652dd24198f57a224e1276935f65b7ddbdb4307c4d0cf4d4fffa4a5798/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/959352dc8af9a86f5ad011d8a7dca62ff010d4f493ec615864ead9b94e211438/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/959352dc8af9a86f5ad011d8a7dca62ff010d4f493ec615864ead9b94e211438/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e11bf61e8d76a916e4d27eb2319598ff97d3177162ff6bef90f5dbf28f19194c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e11bf61e8d76a916e4d27eb2319598ff97d3177162ff6bef90f5dbf28f19194c/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/163e0310-6238-4518-9d5a-c297c56f3603/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/163e0310-6238-4518-9d5a-c297c56f3603/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7e173f-fd22-4319-a168-d22dd2d3ca3b/volumes/kubernetes.io~projected/kube-api-access-5jhqw:{mountpoint:/var/lib/kubelet/pods/3b7e173f-fd22-4319-a168-d22dd2d3ca3b/volumes/kubernetes.io~projected/kube-api-access-5jhqw major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43c945bb-127b-4ce1-9807-dc3de3cb297b/volumes/kubernetes.io~projected/kube-api-access-m4n5d:{mountpoint:/var/lib/kubelet/pods/43c945bb-127b-4ce1-9807-dc3de3cb297b/volumes/kubernetes.io~projected/kube-api-access-m4n5d major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~projected/kube-api-access-b28d9:{mountpoint:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~projected/kube-api-access-b28d9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~projected/kube-api-access-586wm:{mountpoint:/var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~projected/kube-api-access-586wm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b/volumes/kubernetes.io~projected/kube-api-access-h7tbf:{mountpoint:/var/lib/kubelet/pods/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b/volumes/kubernetes.io~projected/kube-api-access-h7tbf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84fedff4-406d-4691-b110-e7b297296f71/volumes/kubernetes.io~projected/kube-api-access-ttn4g:{mountpoint:/var/lib/kubelet/pods/84fedff4-406d-4691-b110-e7b297296f71/volumes/kubernetes.io~projected/kube-api-access-ttn4g major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88cd0b32-9830-43e7-a03d-3dceef10e7ac/volumes/kubernetes.io~projected/kube-api-access-zq6jm:{mountpoint:/var/lib/kubelet/pods/88cd0b32-9830-43e7-a03d-3dceef10e7ac/volumes/kubernetes.io~projected/kube-api-access-zq6jm major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/890281f9-0645-46ac-98d5-93906021aa03/volumes/kubernetes.io~projected/kube-api-access-w796t:{mountpoint:/var/lib/kubelet/pods/890281f9-0645-46ac-98d5-93906021aa03/volumes/kubernetes.io~projected/kube-api-access-w796t major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c88dc4f-1666-4eec-a7cc-17b63c391400/volumes/kubernetes.io~projected/kube-api-access-lgwct:{mountpoint:/var/lib/kubelet/pods/8c88dc4f-1666-4eec-a7cc-17b63c391400/volumes/kubernetes.io~projected/kube-api-access-lgwct major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb7620e-d279-4b5f-863f-a7c5c7820aba/volumes/kubernetes.io~projected/kube-api-access-fsft4:{mountpoint:/var/lib/kubelet/pods/adb7620e-d279-4b5f-863f-a7c5c7820aba/volumes/kubernetes.io~projected/kube-api-access-fsft4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~projected/kube-api-access-w56kd:{mountpoint:/var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~projected/kube-api-access-w56kd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~secret/signing-key major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c745d5c2-e685-49c5-90f5-9b6b0946acca/volumes/kubernetes.io~projected/kube-api-access-8h8v4:{mountpoint:/var/lib/kubelet/pods/c745d5c2-e685-49c5-90f5-9b6b0946acca/volumes/kubernetes.io~projected/kube-api-access-8h8v4 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d61b3b7e-953d-48e5-b08d-792f90f55ce8/volumes/kubernetes.io~projected/kube-api-access-bwct5:{mountpoint:/var/lib/kubelet/pods/d61b3b7e-953d-48e5-b08d-792f90f55ce8/volumes/kubernetes.io~projected/kube-api-access-bwct5 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d80631ef-63c9-400c-a46f-6eed3ca27590/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d80631ef-63c9-400c-a46f-6eed3ca27590/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e47ee884-8fa6-49cc-843e-353afd0c281f/volumes/kubernetes.io~projected/kube-api-access-b574r:{mountpoint:/var/lib/kubelet/pods/e47ee884-8fa6-49cc-843e-353afd0c281f/volumes/kubernetes.io~projected/kube-api-access-b574r major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e75e3379-3433-4e3f-a6a0-52afc2f47330/volumes/kubernetes.io~projected/kube-api-access-9cdqt:{mountpoint:/var/lib/kubelet/pods/e75e3379-3433-4e3f-a6a0-52afc2f47330/volumes/kubernetes.io~projected/kube-api-access-9cdqt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~projected/kube-api-access-hj9nb:{mountpoint:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~projected/kube-api-access-hj9nb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8512c9fe7a29e1c87876c32b936f6e3606e9539169deeef30348c9cb96947a64/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fd1be19a8f25578391c75450a1d7c4ad6646d5eee0aa1b416d578cc958227251/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/abf4d77dd07dff9e80ee66784d1b1685bee213d41263a2bfa59afa9cc9ee008f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/83f4f4dabe4d09d66a5abea9cff491dba9841c16f10d4f018de65e85030b1399/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d3c2433eb4a742068b5e64f304b7d2f55435f39f3fef358e1320e6b3afae121a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ff9446f1da285c0acb2dd1e8a4f040a9b8a846d1df7eaf6ef1fdd235020d52cd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/41f3284fa3494a9fc9b8df1847e7e9ca3bdfe4bd5596c4c35a6abd470e518c99/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c1eed529d1a869fa201e151cb74bb6b202a3712a0b6de64d6f72b1727dd657a8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e37c6db80d2db0305979ca17e71c103f8f36e2decc9b1621bdca3e6f069ce55d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f082d739d581a5287899f68880a8cc46a738e22f8b02fedb7a300aa522d93ed1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d2a1be91486c3395e1eb139badc59de0d7c77cdb786d5a78526f4fc9318c2fb7/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/8375cdba52a581e6d7a8be2392673905771d62fa951b809f389505e627f642ef/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/189b6fc7401c9767e9b57d9a905fdd339d6f7e6ddeb72905028c1a48eac3c1d3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9eb8e428d34e4d833e8343360b601e3c933bdf7dd841dfaf25a20dc102156342/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/453538dd58d08c7cd6238eb9e1adf642204be6fc29f3aacd0d8f29aeb9ce783f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/508a69f05a96296bca4cb7748e1a552d306aa2a1b1fb4d35240e410b19c39130/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cc0e3a4d218cb91a0080e54b4b17557972d4b434f180766ab8c2cb3df23eebe7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f34383936bd3f0e253f69137b0a24fc05876b5e9a71ab2ad57e492758d47740d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/69aa4f8bc79382b71bd46321e0a7b37d8989b009aa24b5aa6486e9e04b978f02/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/419bed884cf923326b07f101bfef1031e13a660e51400e2ea4c00f1fdb10386d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/915c79d643260e1ac0a67438e969f4d3e1164549f10008229e676d437f04437e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/702102b7c00cdd34a2690ba732da818a92c0131a5f58f849e6de4655ae3a11a6/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/066e03601aa2a8270cdca5e621573e7b136d74f5953f1fe5126791d7667bbcb8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/594c4ba77c93b6d12733f9e62f35e4d0b69315ceb92fd115f0554ce5232deaf0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/93c27bc0d9ddcc35793981400741ac3423af717dafe44aa7abaad0225c63f57c/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/94f2fd66271c9d54ddb788a54aef3806cb1537ebb015589ca29afb24c7f1dfa9/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/a9b6e321c073df128473e83ead4eb7972740b102483663f605cfe270cf5d3825/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/96b7365b208c1a215c00f820a58c7a70893cd76b209f563c74d29a6eabc33fe9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/39029995e742d504dcd514222e581ecddd9ef37d11782febefcbfc08abc68ea7/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/07ac25491346205117b5540ca5efe25c9a58eb876fd5453d145d20a9d31229c0/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/a4f6fe24b99cca491500130aa8c1075ba5ddfdd1983975c8c0c7d499c7198fbb/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c08a937d28154c35097285d9c15f1b80cb47470604c73a093f7182e7b1fab490/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/c6bce70b8e1afad1699cd2e9f0e9101d16985d5e18758c1cf8415ea411af5c67/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0b2b04400f1db318ced92ce205781978993efd875197a8622323fda21c4b043a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2e223dcce2e312ff2d77dee8accb1e9ff9791da126070442969c433e8decd757/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2422e2d4d0a90035253761f0fbb2228cf8ba33691aa9e86d6712397fe38806b0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bfa7de8c4722b5c39982e0d2ec30eca58c60fb1b33e3017cc9c590441320ff83/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/52fad6708bbbc2062b199fd7f1e168800065fa080ce115a375854999bedf43c6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/edf0c07d96fe809e6cb5220acdd58a77df17b553f98cbfba2edb3d4b1b5f8298/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/fb92708f4e06ce95b035b45eddab7dd20cabf23fa9b73a0757ecb18a940c381b/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2f242f5c9ff06e7fb8f8b62293b322ba4be4e5080cff646dd747d970a031ec48/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f93d0922fab45317184e699b83a92de026b111bc018eff8a0900b807ae35cc80/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/32cbfde8b785d3a75bfd0c4c56719a5ee12170e0462d9d53a0e461db23b6b5fd/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.841728    4528 manager.go:217] Machine: {Timestamp:2026-10-30 02:40:41.841266261 +0000 UTC m=+0.065515531 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104634 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c59bc033f16b27c6a50b002eb406f SystemUUID:ec2c59bc-033f-16b2-7c6a-50b002eb406f BootID:84f01230-c0f3-49bb-85b0-abf9e56c3faa Filesystems:[{Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea488a745cdc91e2e4fde8e125a688942a0462440466946c2bdaf5d77ab62f96/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e1e191056537d9cbf5aba24684d67820461f0cfa143c5ad6b13de0ed8ab3fff/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21696c652dd24198f57a224e1276935f65b7ddbdb4307c4d0cf4d4fffa4a5798/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d80631ef-63c9-400c-a46f-6eed3ca27590/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e11bf61e8d76a916e4d27eb2319598ff97d3177162ff6bef90f5dbf28f19194c/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~projected/kube-api-access-586wm DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/890281f9-0645-46ac-98d5-93906021aa03/volumes/kubernetes.io~projected/kube-api-access-w796t DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8c88dc4f-1666-4eec-a7cc-17b63c391400/volumes/kubernetes.io~projected/kube-api-access-lgwct DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b/volumes/kubernetes.io~projected/kube-api-access-h7tbf DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e75e3379-3433-4e3f-a6a0-52afc2f47330/volumes/kubernetes.io~projected/kube-api-access-9cdqt DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57872ae63687bcaa6a2b62202ed438239cb3bf492cd70bfd18b82e1865b6527c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/959352dc8af9a86f5ad011d8a7dca62ff010d4f493ec615864ead9b94e211438/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7e173f-fd22-4319-a168-d22dd2d3ca3b/volumes/kubernetes.io~projected/kube-api-access-5jhqw DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43c945bb-127b-4ce1-9807-dc3de3cb297b/volumes/kubernetes.io~projected/kube-api-access-m4n5d DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7116becc5a41f13adf0eb3c05f89363be9ff0dfaac34280425e6f30e47f4b02f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e47ee884-8fa6-49cc-843e-353afd0c281f/volumes/kubernetes.io~projected/kube-api-access-b574r DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/163e0310-6238-4518-9d5a-c297c56f3603/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e39e82a1d2cbb73b7c559d0d2a25fc05276f3326cf4c2b6acaa47a4dfa566f46/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/adb7620e-d279-4b5f-863f-a7c5c7820aba/volumes/kubernetes.io~projected/kube-api-access-fsft4 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/023acfb94290b05e20b04b7eef6878544b2ab0f1b504cd7b02ff7ea322bc8ebb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88cd0b32-9830-43e7-a03d-3dceef10e7ac/volumes/kubernetes.io~projected/kube-api-access-zq6jm DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d61b3b7e-953d-48e5-b08d-792f90f55ce8/volumes/kubernetes.io~projected/kube-api-access-bwct5 DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/84fedff4-406d-4691-b110-e7b297296f71/volumes/kubernetes.io~projected/kube-api-access-ttn4g DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/065badc350da2ad8820ec51b7c63c2e7fb1871d33811559533767b9834092181/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c745d5c2-e685-49c5-90f5-9b6b0946acca/volumes/kubernetes.io~projected/kube-api-access-8h8v4 DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1906ec38e202e36708f7903bc9eba4a0d93bba7f1a057dca013b9866d22af190/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1292202-817e-4114-b11f-91f3e6c65f58/volumes/kubernetes.io~projected/kube-api-access-w56kd DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9682e22749866b67db6e839557ef849e492fb62a18c482982551b0b0afcbe92e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e21900d0247eab7d59d4cfd978ecd6485e89f1c5ab599718d6f38a7182454a3/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/533992c7-201e-4aa7-a5eb-0d956cb8cff8/volumes/kubernetes.io~projected/kube-api-access-b28d9 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51/volumes/kubernetes.io~projected/kube-api-access-hj9nb DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:065badc350da2ad MacAddress:4a:40:cf:02:f6:48 Speed:10000 Mtu:8901} {Name:21696c652dd2419 MacAddress:3a:fc:55:5b:44:b3 Speed:10000 Mtu:8901} {Name:2e21900d0247eab MacAddress:66:47:62:0c:b1:c5 Speed:10000 Mtu:8901} {Name:4e1e191056537d9 MacAddress:86:05:96:7b:d6:b6 Speed:10000 Mtu:8901} {Name:58cfa44f519b051 MacAddress:8e:c3:5b:37:1d:14 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:bd:11:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:46:3a:5a:a4:0e:d2 Speed:0 Mtu:8901} {Name:ea488a745cdc91e MacAddress:b2:1c:c6:fc:87:cf Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:bd:11:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:3a:8b:4d:4a:34:83 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:b5:d8:ba:4c:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.841989    4528 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842031    4528 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842312    4528 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842337    4528 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-164.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842455    4528 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842463    4528 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842476    4528 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842495    4528 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842568    4528 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842729    4528 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842860    4528 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842896    4528 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842909    4528 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842920    4528 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842928    4528 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.842936    4528 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.843506    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.843516    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:41.845247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.843962    4528 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:40:41.846382 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.846300    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:40:41.846432 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.846382    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:40:41.846748 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.846734    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:40:41.847213 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.847177    4528 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:40:41.847279 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.847245    4528 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:40:41.847439 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.847428    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:40:41.847732 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.847700    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:40:41.847798 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.847782    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:40:41.848031 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848010    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:40:41.848063 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848035    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:40:41.848063 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848046    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:40:41.848063 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848055    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:40:41.848150 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848064    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:40:41.848150 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848076    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:40:41.848150 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848085    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:40:41.848150 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848096    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:40:41.848150 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848113    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:40:41.848349 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848155    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:40:41.848349 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.848165    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:40:41.851484 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.851471    4528 server.go:1263] "Started kubelet"
Oct 30 02:40:41.851582 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.851535    4528 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:40:41.851634 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.851562    4528 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:40:41.851681 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.851637    4528 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:40:41.852011 ip-10-0-134-164 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:40:41.852174 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.852024    4528 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:40:41.853592 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.853579    4528 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:40:41.854707 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.854687    4528 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:40:41.855003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.854986    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:40:41.855196 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.855112    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:35:04 +0000 UTC" deadline="2028-06-22 16:31:48.340901623 +0000 UTC"
Oct 30 02:40:41.855196 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.855149    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14437h51m6.48575634s"
Oct 30 02:40:41.855196 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:41.855177    4528 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-164.ec2.internal\" not found"
Oct 30 02:40:41.855373 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.855272    4528 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:40:41.855373 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.855291    4528 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:40:41.855468 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.855428    4528 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:40:41.857928 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.857903    4528 factory.go:55] Registering systemd factory
Oct 30 02:40:41.858020 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.857932    4528 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:40:41.858986 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.858941    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533992c7-201e-4aa7-a5eb-0d956cb8cff8" volumeName="kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.858995    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533992c7-201e-4aa7-a5eb-0d956cb8cff8" volumeName="kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859009    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b" volumeName="kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859026    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cd0b32-9830-43e7-a03d-3dceef10e7ac" volumeName="kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859038    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e75e3379-3433-4e3f-a6a0-52afc2f47330" volumeName="kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859050    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="163e0310-6238-4518-9d5a-c297c56f3603" volumeName="kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs" seLinuxMountContext=""
Oct 30 02:40:41.859074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859063    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7e173f-fd22-4319-a168-d22dd2d3ca3b" volumeName="kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859079    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43c945bb-127b-4ce1-9807-dc3de3cb297b" volumeName="kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859094    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c745d5c2-e685-49c5-90f5-9b6b0946acca" volumeName="kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859114    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8c5bf6-8ec6-4224-a344-4c4da9c21a51" volumeName="kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859126    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8c5bf6-8ec6-4224-a344-4c4da9c21a51" volumeName="kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859137    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8c5bf6-8ec6-4224-a344-4c4da9c21a51" volumeName="kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859150    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7e173f-fd22-4319-a168-d22dd2d3ca3b" volumeName="kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859165    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fedff4-406d-4691-b110-e7b297296f71" volumeName="kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859181    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cd0b32-9830-43e7-a03d-3dceef10e7ac" volumeName="kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859218    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890281f9-0645-46ac-98d5-93906021aa03" volumeName="kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859231    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb7620e-d279-4b5f-863f-a7c5c7820aba" volumeName="kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859255    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="163e0310-6238-4518-9d5a-c297c56f3603" volumeName="kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859278    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43c945bb-127b-4ce1-9807-dc3de3cb297b" volumeName="kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859292    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1292202-817e-4114-b11f-91f3e6c65f58" volumeName="kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859302    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8c5bf6-8ec6-4224-a344-4c4da9c21a51" volumeName="kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859317    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a135dd3-ea50-4c74-86d7-38c5e2d36ca9" volumeName="kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859328    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88cd0b32-9830-43e7-a03d-3dceef10e7ac" volumeName="kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859339    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb7620e-d279-4b5f-863f-a7c5c7820aba" volumeName="kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859351    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb7620e-d279-4b5f-863f-a7c5c7820aba" volumeName="kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859372    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1292202-817e-4114-b11f-91f3e6c65f58" volumeName="kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859385    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1292202-817e-4114-b11f-91f3e6c65f58" volumeName="kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key" seLinuxMountContext=""
Oct 30 02:40:41.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859400    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e47ee884-8fa6-49cc-843e-353afd0c281f" volumeName="kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859424    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8c5bf6-8ec6-4224-a344-4c4da9c21a51" volumeName="kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859436    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43c945bb-127b-4ce1-9807-dc3de3cb297b" volumeName="kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859453    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="533992c7-201e-4aa7-a5eb-0d956cb8cff8" volumeName="kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859467    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e75e3379-3433-4e3f-a6a0-52afc2f47330" volumeName="kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859479    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43c945bb-127b-4ce1-9807-dc3de3cb297b" volumeName="kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859499    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a135dd3-ea50-4c74-86d7-38c5e2d36ca9" volumeName="kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859511    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b" volumeName="kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859521    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fedff4-406d-4691-b110-e7b297296f71" volumeName="kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859532    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="890281f9-0645-46ac-98d5-93906021aa03" volumeName="kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859562    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c88dc4f-1666-4eec-a7cc-17b63c391400" volumeName="kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859574    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb7620e-d279-4b5f-863f-a7c5c7820aba" volumeName="kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859591    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d61b3b7e-953d-48e5-b08d-792f90f55ce8" volumeName="kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859616    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7e173f-fd22-4319-a168-d22dd2d3ca3b" volumeName="kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859629    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84fedff4-406d-4691-b110-e7b297296f71" volumeName="kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859640    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d80631ef-63c9-400c-a46f-6eed3ca27590" volumeName="kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret" seLinuxMountContext=""
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859652    4528 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.859659    4528 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860032    4528 factory.go:153] Registering CRI-O factory
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860046    4528 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860107    4528 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860131    4528 factory.go:103] Registering Raw factory
Oct 30 02:40:41.860433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860221    4528 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:40:41.861358 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.860682    4528 manager.go:319] Starting recovery of all containers
Oct 30 02:40:41.883838 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.883570    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:40:41.886185 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.886148    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:40:41.886185 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.886186    4528 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:40:41.886349 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.886211    4528 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:40:41.886349 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:41.886271    4528 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:40:41.896009 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.895972    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="806fe11cd6bf43b5c870612556231af0ec84c9b4720c65063415f66df24d8ebd" exitCode=0
Oct 30 02:40:41.896009 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.895994    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="81fb1f792789083bf3889f8fb047b9ea4c3026e1b1e1501fa42bc3f4ffb43275" exitCode=0
Oct 30 02:40:41.896009 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.896003    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="b87abfbb6933a9656bf559d640f7e2fc1d44ffcdd6091c95b3cc5269c3ad0209" exitCode=0
Oct 30 02:40:41.896009 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.896011    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="d9423d73997b89f13eb2adae0f1a4c6e802ff94c01eb8d2e3b3f00da4b6b9088" exitCode=0
Oct 30 02:40:41.896259 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.896018    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="a84cea01f4b9f5a9b4ee28e3b1aa9b2eaa61ead0bf4d37f92d404a8121a96e20" exitCode=0
Oct 30 02:40:41.896259 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.896040    4528 generic.go:356] "Generic (PLEG): container finished" podID="adb7620e-d279-4b5f-863f-a7c5c7820aba" containerID="2b9cc97d58b4ca819bda59dfb3a72d2f3af648e04161c87cd66945c12535a25e" exitCode=0
Oct 30 02:40:41.896782 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.896754    4528 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:40:41.897389 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.897367    4528 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:41.897623 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.897598    4528 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:41.908064 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.908039    4528 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:41.911004 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.910980    4528 generic.go:356] "Generic (PLEG): container finished" podID="b87383a6fdf9fa44e793f1fdd2d5341d" containerID="e983218078749d5f7121c9f26876d574c83e1f107522578c48a700b03a5541b5" exitCode=0
Oct 30 02:40:41.911901 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.911877    4528 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:40:41.942414 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.942389    4528 manager.go:324] Recovery completed
Oct 30 02:40:41.957527 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957505    4528 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:40:41.957527 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957520    4528 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:40:41.957707 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957536    4528 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:40:41.957707 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957692    4528 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957704    4528 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957719    4528 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957728    4528 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957736    4528 policy_none.go:50] "Start"
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957746    4528 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:40:41.957769 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.957756    4528 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:40:41.958381 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.958363    4528 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:40:41.958416 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.958382    4528 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:40:41.958416 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.958409    4528 policy_none.go:44] "Start"
Oct 30 02:40:41.962619 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.962564    4528 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:40:41.965242 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965229    4528 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:40:41.965331 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:41.965252    4528 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:40:41.965331 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965260    4528 server.go:87] "Starting device plugin registration server"
Oct 30 02:40:41.965525 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965512    4528 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:40:41.965584 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965524    4528 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:40:41.965637 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965593    4528 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:40:41.965764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965720    4528 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:40:41.965847 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965776    4528 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:40:41.965847 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965785    4528 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:40:41.965946 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.965864    4528 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:40:41.965761559Z","UUID":"879787d8-643c-4141-8bc0-1026c64b67f1","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:40:41.967113 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:41.967091    4528 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:40:41.967447 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.967431    4528 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:40:41.967550 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.967468    4528 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:40:41.986361 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.986319    4528 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"]
Oct 30 02:40:41.986876 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.986845    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerStarted","Data":"f427d4365a4eeced72ca2f1b02d468205eb7ba3d51808d3d325439aabdc2f79e"}
Oct 30 02:40:41.986954 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.986884    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerDied","Data":"e983218078749d5f7121c9f26876d574c83e1f107522578c48a700b03a5541b5"}
Oct 30 02:40:41.986954 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.986900    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:41.987048 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.986905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal" event={"ID":"b87383a6fdf9fa44e793f1fdd2d5341d","Type":"ContainerStarted","Data":"677f34f6e66c81d74dd7f5919e1e914a49b7b3842d6a4d8569dfc9fc34089502"}
Oct 30 02:40:41.987048 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.987031    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"57d17cb448486bb943d1114ac107c4da","Type":"ContainerStarted","Data":"5dcc551edd9aedcc2235fcb695de77787f08906a648cbe9ce38906aa3eb1fd10"}
Oct 30 02:40:41.987140 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.987048    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:41.987140 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:41.987051    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal" event={"ID":"57d17cb448486bb943d1114ac107c4da","Type":"ContainerStarted","Data":"7bec97cc3ebeab4b5b9bb73cca1319e26e26f91911b140eb2391e9c8b6b6ec58"}
Oct 30 02:40:42.014055 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.014027    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:42.014138 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:42.014075    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.020473 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.020457    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:42.020571 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:42.020492    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.066311 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.066281    4528 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:40:42.067666 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.067645    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:40:42.067768 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.067677    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:40:42.067768 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.067691    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-164.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:40:42.067768 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.067725    4528 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.098755 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.098732    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.098878 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.098763    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.098878 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.098781    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.104716 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.104698    4528 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.104775 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.104766    4528 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199365 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199327    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199405    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57d17cb448486bb943d1114ac107c4da-config\") pod \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" (UID: \"57d17cb448486bb943d1114ac107c4da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.199578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.199464    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b87383a6fdf9fa44e793f1fdd2d5341d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" (UID: \"b87383a6fdf9fa44e793f1fdd2d5341d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.843865 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.843830    4528 apiserver.go:52] "Watching apiserver"
Oct 30 02:40:42.887304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.887273    4528 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:40:42.887839 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.887807    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2cr8","openshift-multus/multus-2mzsj","openshift-service-ca/service-ca-5fffc97bdf-2m9bp","kube-system/konnectivity-agent-kgg59","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal","openshift-image-registry/node-ca-nrwb4","openshift-multus/network-metrics-daemon-wdqp2","openshift-network-operator/iptables-alerter-w27r9","openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5","openshift-insights/insights-runtime-extractor-2hqz8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc","openshift-dns/node-resolver-2gm47","openshift-ovn-kubernetes/ovnkube-node-jt8jx","openshift-cluster-node-tuning-operator/tuned-bh7xs","openshift-insights/periodic-gathering-j7nq4-qbr4g","openshift-network-diagnostics/network-check-target-69lbp","kube-system/global-pull-secret-syncer-j4z9f","openshift-dns/dns-default-7kjhk","openshift-multus/multus-additional-cni-plugins-86xxs"]
Oct 30 02:40:42.894028 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.894004    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:42.895016 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.894998    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:42.895829 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.895801    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:42.901549 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901525    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-74jzq\""
Oct 30 02:40:42.901795 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901773    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901803    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7tzfh\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901838    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901842    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901860    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.901859    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902011    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902026    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2trkp\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902029    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:40:42.902056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902045    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:40:42.902486 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902171    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:40:42.902536 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902529    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:40:42.902889 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.902873    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nmhn4\""
Oct 30 02:40:42.904713 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.904693    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2qbtl\""
Oct 30 02:40:42.904713 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.904713    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xkr2x\""
Oct 30 02:40:42.904862 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.904847    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:40:42.905205 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.905188    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.905205 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.905197    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.905308 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.905220    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.933993 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.933972    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.934120 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.934107    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.941189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.941169    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:40:42.941189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.941174    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.941362 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.941214    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.941416 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.941385    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.942909 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.942885    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.943030 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943008    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:40:42.943279 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943174    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.943433 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943333    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.943673 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943643    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.943757 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943718    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.943757 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943738    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943766    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943769    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943769    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943797    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943838    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.943884 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943869    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:40:42.944152 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.943910    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:40:42.944152 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944011    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.944152 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944027    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 02:40:42.944152 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944141    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.944332 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944145    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:42.944332 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944208    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.944332 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944240    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:40:42.944332 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944264    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:40:42.944494 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944353    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:40:42.944494 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944394    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:40:42.944494 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.944411    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:40:42.953523 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.953502    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:42.953613 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:42.953567    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-164.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.954209 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.954190    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:40:42.954382 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.954353    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:40:42.954475 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:42.954439    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-164.ec2.internal"
Oct 30 02:40:42.957537 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:42.957522    4528 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:40:43.001699 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.001672    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:40:43.001807 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.001772    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lz8p4\""
Oct 30 02:40:43.001977 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.001957    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:40:43.002046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.001990    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:40:43.002097 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002064    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ksjrs\""
Oct 30 02:40:43.002171 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002108    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.002171 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002142    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.002171 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002170    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p7m4t\""
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002179    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002197    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8td9f\""
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002232    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002241    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cb5f2\""
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002264    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rv7wz\""
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002286    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84fedff4-406d-4691-b110-e7b297296f71-tmp-dir\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002303    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28d9\" (UniqueName: \"kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002327    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q9zqs\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002335    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002348    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nd26x\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002414    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002438    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002466    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dfk55\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002545    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpstg\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002518    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fd952\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002547    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002606    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-script-lib\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002614    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jvpkf\""
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002625    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cni-binary-copy\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002655    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002692    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002709    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.002733 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002837    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e75e3379-3433-4e3f-a6a0-52afc2f47330-tmp-dir\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002911    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002952    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tbf\" (UniqueName: \"kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002970    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.002984    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003001    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003014    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-daemon-config\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003223    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwct\" (UniqueName: \"kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003321    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.003392 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003402    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003458    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003482    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003481    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-env-overrides\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003575    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003601    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003649    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq6jm\" (UniqueName: \"kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003679    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003905    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.004039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.003937    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56kd\" (UniqueName: \"kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004056    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004173    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004197    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004251    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004302    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004345    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004405    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w796t\" (UniqueName: \"kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.004463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004439    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004460    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004488    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004505    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004550    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586wm\" (UniqueName: \"kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004602    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/890281f9-0645-46ac-98d5-93906021aa03-iptables-alerter-script\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004709    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8v4\" (UniqueName: \"kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4\") pod \"migrator-66df5d496b-t6hs5\" (UID: \"c745d5c2-e685-49c5-90f5-9b6b0946acca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004743    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004753    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1292202-817e-4114-b11f-91f3e6c65f58-signing-key\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.004789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004771    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsft4\" (UniqueName: \"kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004792    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004873    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004895    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004905    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-data-volume\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.004911    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttn4g\" (UniqueName: \"kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005112    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005133    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005152    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.005189 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005189    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005222    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005263    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005291    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005368    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-tuned\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1292202-817e-4114-b11f-91f3e6c65f58-signing-cabundle\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005378    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005528    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005579    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005603    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/163e0310-6238-4518-9d5a-c297c56f3603-agent-certs\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005605    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005632    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovnkube-config\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005663    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005710    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/163e0310-6238-4518-9d5a-c297c56f3603-konnectivity-ca\") pod \"konnectivity-agent-kgg59\" (UID: \"163e0310-6238-4518-9d5a-c297c56f3603\") " pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005716    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005746    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.005762 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005761    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-ovn-node-metrics-cert\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005854    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005891    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005950    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005960    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d80631ef-63c9-400c-a46f-6eed3ca27590-original-pull-secret\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005982    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b574r\" (UniqueName: \"kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.005999    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006013    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006030    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/533992c7-201e-4aa7-a5eb-0d956cb8cff8-tmp\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006087    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006113    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-serviceca\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006118    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006158    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhqw\" (UniqueName: \"kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006300    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdqt\" (UniqueName: \"kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adb7620e-d279-4b5f-863f-a7c5c7820aba-cni-binary-copy\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006401    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006496    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006524    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006550    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.006641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.007329 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006683    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84fedff4-406d-4691-b110-e7b297296f71-config-volume\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.007329 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006699    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.007329 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.006781    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-cert\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:43.022012 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.021989    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tbf\" (UniqueName: \"kubernetes.io/projected/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-kube-api-access-h7tbf\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.022157 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.022138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9nb\" (UniqueName: \"kubernetes.io/projected/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-kube-api-access-hj9nb\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.022203 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.022189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwct\" (UniqueName: \"kubernetes.io/projected/8c88dc4f-1666-4eec-a7cc-17b63c391400-kube-api-access-lgwct\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:43.022320 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.022305    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28d9\" (UniqueName: \"kubernetes.io/projected/533992c7-201e-4aa7-a5eb-0d956cb8cff8-kube-api-access-b28d9\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.028838 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.028793    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttn4g\" (UniqueName: \"kubernetes.io/projected/84fedff4-406d-4691-b110-e7b297296f71-kube-api-access-ttn4g\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.028921 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.028798    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w796t\" (UniqueName: \"kubernetes.io/projected/890281f9-0645-46ac-98d5-93906021aa03-kube-api-access-w796t\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.029158 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029143    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwct5\" (UniqueName: \"kubernetes.io/projected/d61b3b7e-953d-48e5-b08d-792f90f55ce8-kube-api-access-bwct5\") pod \"network-check-target-69lbp\" (UID: \"d61b3b7e-953d-48e5-b08d-792f90f55ce8\") " pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:43.029158 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029148    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq6jm\" (UniqueName: \"kubernetes.io/projected/88cd0b32-9830-43e7-a03d-3dceef10e7ac-kube-api-access-zq6jm\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.029327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029307    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b574r\" (UniqueName: \"kubernetes.io/projected/e47ee884-8fa6-49cc-843e-353afd0c281f-kube-api-access-b574r\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.029400 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029379    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"periodic-gathering-j7nq4-qbr4g\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") " pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:40:43.029720 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029703    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsft4\" (UniqueName: \"kubernetes.io/projected/adb7620e-d279-4b5f-863f-a7c5c7820aba-kube-api-access-fsft4\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.029756 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029706    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdqt\" (UniqueName: \"kubernetes.io/projected/e75e3379-3433-4e3f-a6a0-52afc2f47330-kube-api-access-9cdqt\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.029845 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.029810    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhqw\" (UniqueName: \"kubernetes.io/projected/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-kube-api-access-5jhqw\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.031458 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.031439    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586wm\" (UniqueName: \"kubernetes.io/projected/5a135dd3-ea50-4c74-86d7-38c5e2d36ca9-kube-api-access-586wm\") pod \"ingress-canary-b2cr8\" (UID: \"5a135dd3-ea50-4c74-86d7-38c5e2d36ca9\") " pod="openshift-ingress-canary/ingress-canary-b2cr8"
Oct 30 02:40:43.031595 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.031579    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8v4\" (UniqueName: \"kubernetes.io/projected/c745d5c2-e685-49c5-90f5-9b6b0946acca-kube-api-access-8h8v4\") pod \"migrator-66df5d496b-t6hs5\" (UID: \"c745d5c2-e685-49c5-90f5-9b6b0946acca\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t6hs5"
Oct 30 02:40:43.034027 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.034009    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56kd\" (UniqueName: \"kubernetes.io/projected/c1292202-817e-4114-b11f-91f3e6c65f58-kube-api-access-w56kd\") pod \"service-ca-5fffc97bdf-2m9bp\" (UID: \"c1292202-817e-4114-b11f-91f3e6c65f58\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2m9bp"
Oct 30 02:40:43.090164 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.090126    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:43.107599 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107548    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.107599 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107574    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.107599 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107590    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107604    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107640    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107666    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107674    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-netd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107693    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-dbus\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107705    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107667    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-cnibin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107696    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-cnibin\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107739    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-kubernetes\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.107785 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107760    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-multus\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107796    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107832    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107876    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107885    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b-host\") pod \"node-ca-nrwb4\" (UID: \"5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b\") " pod="openshift-image-registry/node-ca-nrwb4"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107899    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/890281f9-0645-46ac-98d5-93906021aa03-host-slash\") pod \"iptables-alerter-w27r9\" (UID: \"890281f9-0645-46ac-98d5-93906021aa03\") " pod="openshift-network-operator/iptables-alerter-w27r9"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107923    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-k8s-cni-cncf-io\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107901    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-system-cni-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-socket-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107972    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.107987    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108003    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108010    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-modprobe-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108022    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-cni-bin\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108050    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-systemd\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108075    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108081    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-conf-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108097    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108113    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-conf\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108118    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108125    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-systemd\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108131    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108165    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-host\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108162    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108187    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-log-socket\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.108198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108211    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108243    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-etc-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108248    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.608225559 +0000 UTC m=+1.832474838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108263    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-netns\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108279    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108320    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108351    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108401    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysconfig\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108403    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108436    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-etc-sysctl-d\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108437    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108454    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-registration-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108465    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108466    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-etc-kubernetes\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108480    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108498    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-slash\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108499    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108519    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108502    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108536    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adb7620e-d279-4b5f-863f-a7c5c7820aba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86xxs\" (UID: \"adb7620e-d279-4b5f-863f-a7c5c7820aba\") " pod="openshift-multus/multus-additional-cni-plugins-86xxs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108541    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-sys\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108551    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108534    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-var-lib-kubelet\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108589    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.608574321 +0000 UTC m=+1.832823593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108665    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-host-run-multus-certs\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108667    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-kubelet\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108701    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108757    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-run-ovn\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108731    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108781    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108756    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-systemd-units\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.108806    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:43.608791634 +0000 UTC m=+1.833040889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : secret "metrics-daemon-secret" not found
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-device-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108843    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108862    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108876    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108890    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108938    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-node-log\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108936    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108947    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e75e3379-3433-4e3f-a6a0-52afc2f47330-hosts-file\") pod \"node-resolver-2gm47\" (UID: \"e75e3379-3433-4e3f-a6a0-52afc2f47330\") " pod="openshift-dns/node-resolver-2gm47"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-var-lib-kubelet\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108961    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-crio-socket\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108979    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.108995    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109010    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-cni-bin\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109026    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109046    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109055    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e47ee884-8fa6-49cc-843e-353afd0c281f-sys-fs\") pod \"aws-ebs-csi-driver-node-jd5gc\" (UID: \"e47ee884-8fa6-49cc-843e-353afd0c281f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jd5gc"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109061    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d80631ef-63c9-400c-a46f-6eed3ca27590-kubelet-config\") pod \"global-pull-secret-syncer-j4z9f\" (UID: \"d80631ef-63c9-400c-a46f-6eed3ca27590\") " pod="kube-system/global-pull-secret-syncer-j4z9f"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109092    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109109    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109110    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-lib-modules\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109115    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-multus-socket-dir-parent\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109138    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/533992c7-201e-4aa7-a5eb-0d956cb8cff8-run\") pod \"tuned-bh7xs\" (UID: \"533992c7-201e-4aa7-a5eb-0d956cb8cff8\") " pod="openshift-cluster-node-tuning-operator/tuned-bh7xs"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109119    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-system-cni-dir\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109133    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109163    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.109130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109167    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-host-run-netns\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.111135 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88cd0b32-9830-43e7-a03d-3dceef10e7ac-hostroot\") pod \"multus-2mzsj\" (UID: \"88cd0b32-9830-43e7-a03d-3dceef10e7ac\") " pod="openshift-multus/multus-2mzsj"
Oct 30 02:40:43.111135 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.109201    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb8c5bf6-8ec6-4224-a344-4c4da9c21a51-var-lib-openvswitch\") pod \"ovnkube-node-jt8jx\" (UID: \"eb8c5bf6-8ec6-4224-a344-4c4da9c21a51\") " pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:43.480277 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.480241    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:43.612253 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.612218    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:43.612434 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.612263    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:43.612434 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612378    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.612434 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.612386    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:43.612560 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612384    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:43.612560 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612474    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:43.612560 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612455    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls podName:84fedff4-406d-4691-b110-e7b297296f71 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.612421076 +0000 UTC m=+2.836670345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls") pod "dns-default-7kjhk" (UID: "84fedff4-406d-4691-b110-e7b297296f71") : secret "dns-default-metrics-tls" not found
Oct 30 02:40:43.612560 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612534    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.612520869 +0000 UTC m=+2.836770133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:43.612560 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:43.612552    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:44.612540595 +0000 UTC m=+2.836789856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : secret "metrics-daemon-secret" not found
Oct 30 02:40:43.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.617884    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:43.762424 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.762340    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:43.904267 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:43.904234    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:44.297239 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.297203    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:44.609485 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.609416    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:44.619302 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.619280    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:44.619429 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.619314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:44.619429 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.619343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:44.619517 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:44.619443    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:44.619517 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:44.619453    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:44.619517 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:44.619515    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.619498608 +0000 UTC m=+4.843747868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:44.619624 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:44.619532    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:46.619523445 +0000 UTC m=+4.843772701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : secret "metrics-daemon-secret" not found
Oct 30 02:40:44.619703 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.619685    4528 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:40:44.621492 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.621472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84fedff4-406d-4691-b110-e7b297296f71-metrics-tls\") pod \"dns-default-7kjhk\" (UID: \"84fedff4-406d-4691-b110-e7b297296f71\") " pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:44.695692 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.695665    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:44.901962 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:44.901921    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:45.033488 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.033459    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7kjhk"]
Oct 30 02:40:45.042253 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:45.042226    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fedff4_406d_4691_b110_e7b297296f71.slice/crio-66ebf2d8b3a722b3c9d3e0f67ed2ce3b318738f660fddec31be6ef2c175fab29 WatchSource:0}: Error finding container 66ebf2d8b3a722b3c9d3e0f67ed2ce3b318738f660fddec31be6ef2c175fab29: Status 404 returned error can't find the container with id 66ebf2d8b3a722b3c9d3e0f67ed2ce3b318738f660fddec31be6ef2c175fab29
Oct 30 02:40:45.044413 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.044395    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:40:45.103169 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.103128    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:45.498312 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.498274    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:45.701635 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.701603    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:45.942697 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:45.942659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kjhk" event={"ID":"84fedff4-406d-4691-b110-e7b297296f71","Type":"ContainerStarted","Data":"66ebf2d8b3a722b3c9d3e0f67ed2ce3b318738f660fddec31be6ef2c175fab29"}
Oct 30 02:40:46.012017 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.011991    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:46.012251 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.012230    4528 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:40:46.297584 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.297509    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:46.636854 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.636803    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:46.637029 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.636869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:46.637029 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:46.636923    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:46.637029 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:46.636986    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls podName:3b7e173f-fd22-4319-a168-d22dd2d3ca3b nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.636971631 +0000 UTC m=+8.861220907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2hqz8" (UID: "3b7e173f-fd22-4319-a168-d22dd2d3ca3b") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:40:46.637029 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:46.636995    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:40:46.637251 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:46.637034    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs podName:8c88dc4f-1666-4eec-a7cc-17b63c391400 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:50.637024206 +0000 UTC m=+8.861273477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs") pod "network-metrics-daemon-wdqp2" (UID: "8c88dc4f-1666-4eec-a7cc-17b63c391400") : secret "metrics-daemon-secret" not found
Oct 30 02:40:46.658696 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.658670    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jt8jx"
Oct 30 02:40:46.909172 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.909138    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:46.948331 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.948289    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kjhk" event={"ID":"84fedff4-406d-4691-b110-e7b297296f71","Type":"ContainerStarted","Data":"63ab141dfc53775e20b63ff405c7a61d9044f4475f511851c86bd8936b7e80b3"}
Oct 30 02:40:46.948481 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.948356    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7kjhk" event={"ID":"84fedff4-406d-4691-b110-e7b297296f71","Type":"ContainerStarted","Data":"ff4dd3214efaa6a317a3a60b2cb4fd7ea17912742195771519945d661817c835"}
Oct 30 02:40:46.948697 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:46.948679    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:47.104107 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:47.104074    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:47.510051 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:47.510016    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:47.610594 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:47.610566    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kgg59"
Oct 30 02:40:47.705855 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:47.705795    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:48.297360 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:48.297331    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:48.896116 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:48.896075    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.025547 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:49.025516    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.159387 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:49.159304    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.513555 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:49.513465    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.705907 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:49.705874    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:49.893490 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:49.893457    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:50.664679 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.664646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:50.665061 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.664686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:50.666966 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.666945    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b7e173f-fd22-4319-a168-d22dd2d3ca3b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hqz8\" (UID: \"3b7e173f-fd22-4319-a168-d22dd2d3ca3b\") " pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:50.667061 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.666988    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c88dc4f-1666-4eec-a7cc-17b63c391400-metrics-certs\") pod \"network-metrics-daemon-wdqp2\" (UID: \"8c88dc4f-1666-4eec-a7cc-17b63c391400\") " pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:50.694945 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.694916    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdqp2"
Oct 30 02:40:50.697070 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:50.697053    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hqz8"
Oct 30 02:40:51.022945 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.022873    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdqp2"]
Oct 30 02:40:51.028121 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:51.028097    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88dc4f_1666_4eec_a7cc_17b63c391400.slice/crio-aef38dcbfcf2b6688d4679258f9dc22ae9b58ac507b9cc0d0d3ddb297f2bc9b7 WatchSource:0}: Error finding container aef38dcbfcf2b6688d4679258f9dc22ae9b58ac507b9cc0d0d3ddb297f2bc9b7: Status 404 returned error can't find the container with id aef38dcbfcf2b6688d4679258f9dc22ae9b58ac507b9cc0d0d3ddb297f2bc9b7
Oct 30 02:40:51.033038 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.033016    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hqz8"]
Oct 30 02:40:51.041119 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:51.041095    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7e173f_fd22_4319_a168_d22dd2d3ca3b.slice/crio-718116c056c850838a89e2a382ab294c94d508e9239bb81337fbe502ea7b34f0 WatchSource:0}: Error finding container 718116c056c850838a89e2a382ab294c94d508e9239bb81337fbe502ea7b34f0: Status 404 returned error can't find the container with id 718116c056c850838a89e2a382ab294c94d508e9239bb81337fbe502ea7b34f0
Oct 30 02:40:51.594227 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.594197    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:40:51.969342 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.969305    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdqp2" event={"ID":"8c88dc4f-1666-4eec-a7cc-17b63c391400","Type":"ContainerStarted","Data":"14a2c185c3f15b31f48361dee06a0d321253454f08062394944f0f6ca210ea1c"}
Oct 30 02:40:51.969772 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.969407    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdqp2" event={"ID":"8c88dc4f-1666-4eec-a7cc-17b63c391400","Type":"ContainerStarted","Data":"aef38dcbfcf2b6688d4679258f9dc22ae9b58ac507b9cc0d0d3ddb297f2bc9b7"}
Oct 30 02:40:51.971358 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.971317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hqz8" event={"ID":"3b7e173f-fd22-4319-a168-d22dd2d3ca3b","Type":"ContainerStarted","Data":"c5dc7a998d3dc0a6ae8ed8df9ff0fe90f1f9fb35ea30bd1fd40a1afe748661a5"}
Oct 30 02:40:51.971483 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.971378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hqz8" event={"ID":"3b7e173f-fd22-4319-a168-d22dd2d3ca3b","Type":"ContainerStarted","Data":"5eb9c2603a34d16dc0c3f23475819480dd8c76c2b50189d1305c1a5a36aa926c"}
Oct 30 02:40:51.971483 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:51.971407    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hqz8" event={"ID":"3b7e173f-fd22-4319-a168-d22dd2d3ca3b","Type":"ContainerStarted","Data":"718116c056c850838a89e2a382ab294c94d508e9239bb81337fbe502ea7b34f0"}
Oct 30 02:40:52.034924 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.034890    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.166168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.166134    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.289180 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.289149    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.411755 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.411722    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.593074 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.592996    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:52.976941 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:52.976906    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdqp2" event={"ID":"8c88dc4f-1666-4eec-a7cc-17b63c391400","Type":"ContainerStarted","Data":"3d91af81c4e38ea2429c1b4d6272e51fcbfbd04f0bf40ffe4b6dbfe6b345af39"}
Oct 30 02:40:53.246027 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:53.245942    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.360685 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:53.360657    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:53.982168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:53.982126    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hqz8" event={"ID":"3b7e173f-fd22-4319-a168-d22dd2d3ca3b","Type":"ContainerStarted","Data":"eb237972d53d3558994e1aa0b0ca2e2354bd2cb14f6be0dc0142427c8cda6d48"}
Oct 30 02:40:57.588671 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.588634    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vg56v"]
Oct 30 02:40:57.592198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.592171    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.619146 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.619122    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:40:57.619271 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.619231    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:40:57.621215 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.621195    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-s4czt\""
Oct 30 02:40:57.629423 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.629397    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vg56v"]
Oct 30 02:40:57.683140 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.683105    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54fc56d44b-zq9ss"]
Oct 30 02:40:57.686230 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.686210    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.714081 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.714054    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74047d51-f1c2-486d-814d-66b8da177110-tmp\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.714218 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.714093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvpv\" (UniqueName: \"kubernetes.io/projected/74047d51-f1c2-486d-814d-66b8da177110-kube-api-access-wwvpv\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.726880 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.726858    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:40:57.729216 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.729202    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:40:57.729323 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.729256    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:40:57.730788 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.730775    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:40:57.730842 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.730833    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pjcb2\""
Oct 30 02:40:57.773604 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.773577    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54fc56d44b-zq9ss"]
Oct 30 02:40:57.785837 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.785796    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.814507 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814471    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-certificates\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814637 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814512    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-installation-pull-secrets\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814637 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814600    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-image-registry-private-configuration\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814637 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814632    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d77c483c-9cdd-4c94-b087-27c8bc611434-ca-trust-extracted\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814751 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814709    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-bound-sa-token\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814787 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-trusted-ca\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814868 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814789    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74047d51-f1c2-486d-814d-66b8da177110-tmp\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.814929 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-tls\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.814929 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvpv\" (UniqueName: \"kubernetes.io/projected/74047d51-f1c2-486d-814d-66b8da177110-kube-api-access-wwvpv\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.815016 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.814965    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljz5d\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-kube-api-access-ljz5d\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.815153 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.815137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74047d51-f1c2-486d-814d-66b8da177110-tmp\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.840370 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.840317    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvpv\" (UniqueName: \"kubernetes.io/projected/74047d51-f1c2-486d-814d-66b8da177110-kube-api-access-wwvpv\") pod \"downloads-6dd789c8b7-vg56v\" (UID: \"74047d51-f1c2-486d-814d-66b8da177110\") " pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.903934 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.903903    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:40:57.916061 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916034    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-tls\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916069    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljz5d\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-kube-api-access-ljz5d\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-certificates\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916106    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-installation-pull-secrets\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916142    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-image-registry-private-configuration\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916167    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d77c483c-9cdd-4c94-b087-27c8bc611434-ca-trust-extracted\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916424 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916209    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-bound-sa-token\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916424 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916239    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-trusted-ca\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.916974 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.916946    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d77c483c-9cdd-4c94-b087-27c8bc611434-ca-trust-extracted\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.917537 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.917514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-certificates\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.917968 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.917945    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d77c483c-9cdd-4c94-b087-27c8bc611434-trusted-ca\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.919128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.919101    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-image-registry-private-configuration\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.919353 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.919335    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-registry-tls\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.919603 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.919574    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d77c483c-9cdd-4c94-b087-27c8bc611434-installation-pull-secrets\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.937373 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.937343    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-bound-sa-token\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.941566 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.941545    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljz5d\" (UniqueName: \"kubernetes.io/projected/d77c483c-9cdd-4c94-b087-27c8bc611434-kube-api-access-ljz5d\") pod \"image-registry-54fc56d44b-zq9ss\" (UID: \"d77c483c-9cdd-4c94-b087-27c8bc611434\") " pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:57.958377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.958352    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7kjhk"
Oct 30 02:40:57.978267 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.978234    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:57.997062 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:57.997008    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:58.099782 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.099713    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:58.211670 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.211636    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vg56v"]
Oct 30 02:40:58.221116 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:58.221086    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74047d51_f1c2_486d_814d_66b8da177110.slice/crio-b701978c8a6ac36be90103c471e40ecd2ef2908deca72621eae85bedb519a848 WatchSource:0}: Error finding container b701978c8a6ac36be90103c471e40ecd2ef2908deca72621eae85bedb519a848: Status 404 returned error can't find the container with id b701978c8a6ac36be90103c471e40ecd2ef2908deca72621eae85bedb519a848
Oct 30 02:40:58.298056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.298027    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54fc56d44b-zq9ss"]
Oct 30 02:40:58.303136 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:40:58.303109    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77c483c_9cdd_4c94_b087_27c8bc611434.slice/crio-a3de5537dc15e78804fb4bd889cabc3bafbed019164f130fd8c51ebd55499550 WatchSource:0}: Error finding container a3de5537dc15e78804fb4bd889cabc3bafbed019164f130fd8c51ebd55499550: Status 404 returned error can't find the container with id a3de5537dc15e78804fb4bd889cabc3bafbed019164f130fd8c51ebd55499550
Oct 30 02:40:58.812546 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.812513    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:58.951689 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.951640    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:58.999780 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.999739    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss" event={"ID":"d77c483c-9cdd-4c94-b087-27c8bc611434","Type":"ContainerStarted","Data":"899b0da08f254e69be641563dea318fad1b81ae7ea58b279c90b287fe1687c4d"}
Oct 30 02:40:58.999982 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:58.999797    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss" event={"ID":"d77c483c-9cdd-4c94-b087-27c8bc611434","Type":"ContainerStarted","Data":"a3de5537dc15e78804fb4bd889cabc3bafbed019164f130fd8c51ebd55499550"}
Oct 30 02:40:59.000577 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.000552    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:40:59.001958 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.001935    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vg56v" event={"ID":"74047d51-f1c2-486d-814d-66b8da177110","Type":"ContainerStarted","Data":"b701978c8a6ac36be90103c471e40ecd2ef2908deca72621eae85bedb519a848"}
Oct 30 02:40:59.042203 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.042141    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss" podStartSLOduration=2.042121919 podStartE2EDuration="2.042121919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:40:59.041567587 +0000 UTC m=+17.265816864" watchObservedRunningTime="2026-10-30 02:40:59.042121919 +0000 UTC m=+17.266371197"
Oct 30 02:40:59.234033 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.234004    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"]
Oct 30 02:40:59.239722 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.239695    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:40:59.252399 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.252375    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 02:40:59.252539 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.252467    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-glvs5\""
Oct 30 02:40:59.258040 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.258010    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"]
Oct 30 02:40:59.258725 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.258700    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.327670 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.327623    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5g6sc\" (UID: \"adf40128-5210-4cf0-a318-14202b488553\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:40:59.428240 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.428203    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5g6sc\" (UID: \"adf40128-5210-4cf0-a318-14202b488553\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:40:59.428427 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:59.428356    4528 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 02:40:59.428478 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:40:59.428442    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates podName:adf40128-5210-4cf0-a318-14202b488553 nodeName:}" failed. No retries permitted until 2026-10-30 02:40:59.928420521 +0000 UTC m=+18.152669792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-5g6sc" (UID: "adf40128-5210-4cf0-a318-14202b488553") : secret "prometheus-operator-admission-webhook-tls" not found
Oct 30 02:40:59.595195 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.595108    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:40:59.933136 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.933098    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5g6sc\" (UID: \"adf40128-5210-4cf0-a318-14202b488553\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:40:59.939393 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:40:59.939364    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adf40128-5210-4cf0-a318-14202b488553-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5g6sc\" (UID: \"adf40128-5210-4cf0-a318-14202b488553\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:41:00.153246 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:00.153206    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:41:00.453420 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:00.453387    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"]
Oct 30 02:41:00.459292 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:00.459226    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf40128_5210_4cf0_a318_14202b488553.slice/crio-0e0692701b9ef1cad2831eaf646ba1494a41e760e3dfbee649a46021d30a0803 WatchSource:0}: Error finding container 0e0692701b9ef1cad2831eaf646ba1494a41e760e3dfbee649a46021d30a0803: Status 404 returned error can't find the container with id 0e0692701b9ef1cad2831eaf646ba1494a41e760e3dfbee649a46021d30a0803
Oct 30 02:41:01.011875 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:01.011837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc" event={"ID":"adf40128-5210-4cf0-a318-14202b488553","Type":"ContainerStarted","Data":"0e0692701b9ef1cad2831eaf646ba1494a41e760e3dfbee649a46021d30a0803"}
Oct 30 02:41:02.016976 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.016940    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc" event={"ID":"adf40128-5210-4cf0-a318-14202b488553","Type":"ContainerStarted","Data":"f42b4c464f7b65bec2d9306dbe92c6b9a581be9ae702f01e8ef17756b9f418ee"}
Oct 30 02:41:02.017430 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.017134    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:41:02.023302 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.023276    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc"
Oct 30 02:41:02.054057 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.054012    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5g6sc" podStartSLOduration=2.161933762 podStartE2EDuration="3.053998362s" totalImagesPullingTime="892.0646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:02.053774047 +0000 UTC m=+20.278023326" watchObservedRunningTime="2026-10-30 02:41:02.053998362 +0000 UTC m=+20.278247633"
Oct 30 02:41:02.365585 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.365499    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.452880 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.452841    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"]
Oct 30 02:41:02.456481 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.456453    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.472160 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.472127    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:41:02.472910 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.472887    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:41:02.473174 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.473152    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:41:02.482198 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.482171    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 02:41:02.482540 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.482371    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 02:41:02.482540 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.482509    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nwndq\""
Oct 30 02:41:02.483242 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.483207    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"]
Oct 30 02:41:02.528171 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.528137    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.554213 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.554174    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.554383 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.554220    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2tj\" (UniqueName: \"kubernetes.io/projected/cd87f820-c610-4549-b9ea-235e07dd0810-kube-api-access-ll2tj\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.554383 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.554354    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.554457 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.554399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd87f820-c610-4549-b9ea-235e07dd0810-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.655543 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.655504    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.655726 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.655555    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2tj\" (UniqueName: \"kubernetes.io/projected/cd87f820-c610-4549-b9ea-235e07dd0810-kube-api-access-ll2tj\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.655726 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.655598    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.655726 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.655625    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd87f820-c610-4549-b9ea-235e07dd0810-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.656459 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.656418    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd87f820-c610-4549-b9ea-235e07dd0810-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.659520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.659489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.659750 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.659728    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd87f820-c610-4549-b9ea-235e07dd0810-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.662776 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.662738    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.678713 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.678682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2tj\" (UniqueName: \"kubernetes.io/projected/cd87f820-c610-4549-b9ea-235e07dd0810-kube-api-access-ll2tj\") pod \"prometheus-operator-9cd89b74c-szpz7\" (UID: \"cd87f820-c610-4549-b9ea-235e07dd0810\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.772309 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.772274    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"
Oct 30 02:41:02.803949 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.803920    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:02.807227 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.807194    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.812170 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.811807    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:02.820225 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.820201    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:41:02.820225 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.820206    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kclfx\""
Oct 30 02:41:02.820395 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.820235    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:41:02.820995 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.820978    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:41:02.821128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.821110    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:41:02.821128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.821124    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:41:02.835739 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.835632    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:02.958059 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.957980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958059 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958033    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958132    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958164    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdjn\" (UniqueName: \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958247 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958393 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958263    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:02.958393 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:02.958282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.059662 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059629    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059674    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.059969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdjn\" (UniqueName: \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060492 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.060233    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.060492 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.060434    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.061007 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.060985    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.061483 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.061464    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.062654 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.062629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.062999 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.062983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.074594 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.074566    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-szpz7"]
Oct 30 02:41:03.080038 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:03.080013    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd87f820_c610_4549_b9ea_235e07dd0810.slice/crio-12166f90e59266ba86d5723ca7d16e0d1e735d0a845bab50030313df89731cae WatchSource:0}: Error finding container 12166f90e59266ba86d5723ca7d16e0d1e735d0a845bab50030313df89731cae: Status 404 returned error can't find the container with id 12166f90e59266ba86d5723ca7d16e0d1e735d0a845bab50030313df89731cae
Oct 30 02:41:03.085422 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.085398    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdjn\" (UniqueName: \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\") pod \"console-856b7bc648-r699g\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") " pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.121348 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.121312    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:03.428793 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:03.428752    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:03.436084 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:03.436055    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad7dc91_5791_4de9_bcdd_e16b537c93df.slice/crio-89d29217c1d051e0a1cca4b90b09b0e786e574900e914b8a434b8b81dbe6f1d5 WatchSource:0}: Error finding container 89d29217c1d051e0a1cca4b90b09b0e786e574900e914b8a434b8b81dbe6f1d5: Status 404 returned error can't find the container with id 89d29217c1d051e0a1cca4b90b09b0e786e574900e914b8a434b8b81dbe6f1d5
Oct 30 02:41:04.026277 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:04.026185    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856b7bc648-r699g" event={"ID":"2ad7dc91-5791-4de9-bcdd-e16b537c93df","Type":"ContainerStarted","Data":"89d29217c1d051e0a1cca4b90b09b0e786e574900e914b8a434b8b81dbe6f1d5"}
Oct 30 02:41:04.028436 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:04.028378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7" event={"ID":"cd87f820-c610-4549-b9ea-235e07dd0810","Type":"ContainerStarted","Data":"12166f90e59266ba86d5723ca7d16e0d1e735d0a845bab50030313df89731cae"}
Oct 30 02:41:04.640231 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:41:04.640190    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:41:05.035287 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:05.035205    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7" event={"ID":"cd87f820-c610-4549-b9ea-235e07dd0810","Type":"ContainerStarted","Data":"b987fc1b1c3b7d48bb3ad0f8957e32d765e56ce5658728e0686f8ad944e2c762"}
Oct 30 02:41:05.035287 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:05.035268    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7" event={"ID":"cd87f820-c610-4549-b9ea-235e07dd0810","Type":"ContainerStarted","Data":"7d05cea77d0eb87aaff0e4bd332077c80573672eb699f2d90c0bc1ec3bea9b4a"}
Oct 30 02:41:05.072521 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:05.072467    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-szpz7" podStartSLOduration=1.832763994 podStartE2EDuration="3.072448129s" totalImagesPullingTime="1.239684135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:05.071725834 +0000 UTC m=+23.295975111" watchObservedRunningTime="2026-10-30 02:41:05.072448129 +0000 UTC m=+23.296697408"
Oct 30 02:41:05.234004 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:05.233957    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:06.824416 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:06.824386    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.044749 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.044714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856b7bc648-r699g" event={"ID":"2ad7dc91-5791-4de9-bcdd-e16b537c93df","Type":"ContainerStarted","Data":"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"}
Oct 30 02:41:07.046735 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.046700    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.084809 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.084715    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-856b7bc648-r699g" podStartSLOduration=1.708633192 podStartE2EDuration="5.084701035s" totalImagesPullingTime="3.376067843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:07.083974044 +0000 UTC m=+25.308223324" watchObservedRunningTime="2026-10-30 02:41:07.084701035 +0000 UTC m=+25.308950313"
Oct 30 02:41:07.159784 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.159748    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9r8z9"]
Oct 30 02:41:07.178710 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.178683    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.200490 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.200456    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.213130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.213104    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:41:07.213476 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.213456    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:41:07.214083 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.214062    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mmm8g\""
Oct 30 02:41:07.218056 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.218035    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:41:07.297127 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297088    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-tls\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297127 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297137    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-root\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297364 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297219    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-wtmp\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297364 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4nj\" (UniqueName: \"kubernetes.io/projected/c6eb06d5-d557-4d8b-b61e-4c28f8140543-kube-api-access-7q4nj\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297364 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297274    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-accelerators-collector-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297482 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297400    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-metrics-client-ca\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297526 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297503    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-textfile\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297564 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297533    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-sys\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.297600 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.297560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.321870 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.321831    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.398535 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398499    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-textfile\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398547    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-sys\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398573    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398619    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-tls\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398644    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-root\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398666    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-wtmp\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398691    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4nj\" (UniqueName: \"kubernetes.io/projected/c6eb06d5-d557-4d8b-b61e-4c28f8140543-kube-api-access-7q4nj\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.398734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398722    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-accelerators-collector-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398765    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-metrics-client-ca\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.398940    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-textfile\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.399017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-root\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.399058    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-sys\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399390 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.399345    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-metrics-client-ca\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399986 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.399481    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-wtmp\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.399986 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.399945    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-accelerators-collector-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.402571 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.402522    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-tls\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.402865 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.402843    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6eb06d5-d557-4d8b-b61e-4c28f8140543-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.421633 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.421600    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4nj\" (UniqueName: \"kubernetes.io/projected/c6eb06d5-d557-4d8b-b61e-4c28f8140543-kube-api-access-7q4nj\") pod \"node-exporter-9r8z9\" (UID: \"c6eb06d5-d557-4d8b-b61e-4c28f8140543\") " pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.441510 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.441471    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.542262 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.542224    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9r8z9"
Oct 30 02:41:07.564775 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.564740    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.572366 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:07.572338    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eb06d5_d557_4d8b_b61e_4c28f8140543.slice/crio-37cb9f54abeaf1d05c053309e93dcc0764b3131b86f42c14a7d6177f069fa088 WatchSource:0}: Error finding container 37cb9f54abeaf1d05c053309e93dcc0764b3131b86f42c14a7d6177f069fa088: Status 404 returned error can't find the container with id 37cb9f54abeaf1d05c053309e93dcc0764b3131b86f42c14a7d6177f069fa088
Oct 30 02:41:07.691130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.691042    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:07.819704 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:07.819666    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:08.050183 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:08.050102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9r8z9" event={"ID":"c6eb06d5-d557-4d8b-b61e-4c28f8140543","Type":"ContainerStarted","Data":"37cb9f54abeaf1d05c053309e93dcc0764b3131b86f42c14a7d6177f069fa088"}
Oct 30 02:41:08.281225 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:08.281187    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:08.446366 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:08.446328    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:08.587245 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:08.587201    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:09.018386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:09.018361    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:09.218173 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:09.218108    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:09.563467 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:09.563386    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:09.682635 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:09.682598    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:10.061712 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:10.061677    4528 generic.go:356] "Generic (PLEG): container finished" podID="c6eb06d5-d557-4d8b-b61e-4c28f8140543" containerID="df216c5bf55dd40d5164cdb44f06944f46a7181e47ec2bafe85b774569e32216" exitCode=0
Oct 30 02:41:10.061932 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:10.061780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9r8z9" event={"ID":"c6eb06d5-d557-4d8b-b61e-4c28f8140543","Type":"ContainerDied","Data":"df216c5bf55dd40d5164cdb44f06944f46a7181e47ec2bafe85b774569e32216"}
Oct 30 02:41:10.436678 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:10.436641    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:10.601965 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:10.601925    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:10.744643 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:10.744556    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:11.068307 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.068218    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9r8z9" event={"ID":"c6eb06d5-d557-4d8b-b61e-4c28f8140543","Type":"ContainerStarted","Data":"21b9aa60e48db7c304f0ec62b99f0e2770b5ad12fc019b38391c06e2514ed96e"}
Oct 30 02:41:11.068466 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.068316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9r8z9" event={"ID":"c6eb06d5-d557-4d8b-b61e-4c28f8140543","Type":"ContainerStarted","Data":"6806a5dc1c6711aee920abb4749078e508285fa9a515d6e7a4d2bbcc242bfaa5"}
Oct 30 02:41:11.133161 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.133109    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9r8z9" podStartSLOduration=2.660888242 podStartE2EDuration="4.133095196s" totalImagesPullingTime="1.387770027s" totalInitContainerRuntime="84.436927ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:11.12263655 +0000 UTC m=+29.346885830" watchObservedRunningTime="2026-10-30 02:41:11.133095196 +0000 UTC m=+29.357344472"
Oct 30 02:41:11.453722 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.453691    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:11.581812 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.581778    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:11.597910 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.597882    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-69lbp"
Oct 30 02:41:11.709516 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.709423    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:11.929022 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:11.928971    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:12.052503 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:12.052416    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:12.187052 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:12.187023    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:13.121681 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:13.121645    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:13.122714 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:13.122688    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:13.128613 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:13.128591    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:13.847630 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:13.847592    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:14.050186 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.050130    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:14.084292 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.084262    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:14.088060 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.088035    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:14.147694 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.146599    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:41:14.150418 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.150309    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.192226 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.192194    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:41:14.193971 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.193518    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:41:14.205543 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.205514    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:14.256097 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256066    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256097 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256100    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256125    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256267    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256387 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256334    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxv7\" (UniqueName: \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256387 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256376    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256465 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256404    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.256465 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.256439    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.357756 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357716    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.357756 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357803    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357854    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357892    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357954    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxv7\" (UniqueName: \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358003 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.357986    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358647 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.358618    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.358948 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.358921    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.359373 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.359350    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.359947 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.359905    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.360809 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.360778    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.361216 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.361189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.366401 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.366355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.389137 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.389111    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxv7\" (UniqueName: \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\") pod \"console-7c9ccb594d-v6l8z\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") " pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.466907 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.466811    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:14.785694 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:14.785664    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:41:14.791917 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:14.791883    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a166b83_a7d0_4fca_b729_e6a989558098.slice/crio-a57d76c939b07097cc92f6c6dc5f2d77e960322936b6e5de5cc79a0dae1cfbd4 WatchSource:0}: Error finding container a57d76c939b07097cc92f6c6dc5f2d77e960322936b6e5de5cc79a0dae1cfbd4: Status 404 returned error can't find the container with id a57d76c939b07097cc92f6c6dc5f2d77e960322936b6e5de5cc79a0dae1cfbd4
Oct 30 02:41:15.086441 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.086240    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vg56v" event={"ID":"74047d51-f1c2-486d-814d-66b8da177110","Type":"ContainerStarted","Data":"ca859a225ca12145b0c1f01c5d3f17f632e5db91cc0f77840e5b8b9a703864a6"}
Oct 30 02:41:15.086441 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.086319    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:41:15.089422 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.089393    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9ccb594d-v6l8z" event={"ID":"7a166b83-a7d0-4fca-b729-e6a989558098","Type":"ContainerStarted","Data":"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"}
Oct 30 02:41:15.089546 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.089440    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9ccb594d-v6l8z" event={"ID":"7a166b83-a7d0-4fca-b729-e6a989558098","Type":"ContainerStarted","Data":"a57d76c939b07097cc92f6c6dc5f2d77e960322936b6e5de5cc79a0dae1cfbd4"}
Oct 30 02:41:15.106173 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.106145    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vg56v"
Oct 30 02:41:15.123025 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.122967    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-vg56v" podStartSLOduration=1.686282735 podStartE2EDuration="18.12295223s" totalImagesPullingTime="16.436669495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:15.122682414 +0000 UTC m=+33.346931692" watchObservedRunningTime="2026-10-30 02:41:15.12295223 +0000 UTC m=+33.347201506"
Oct 30 02:41:15.210511 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:15.210446    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c9ccb594d-v6l8z" podStartSLOduration=1.210431175 podStartE2EDuration="1.210431175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:15.208497345 +0000 UTC m=+33.432746644" watchObservedRunningTime="2026-10-30 02:41:15.210431175 +0000 UTC m=+33.434680519"
Oct 30 02:41:16.257304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:16.257263    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:16.389030 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:16.388987    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:17.293672 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:17.293634    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:17.428928 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:17.428889    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:18.480847 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:18.480590    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.692834 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:20.692791    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:20.815643 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:20.815604    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.017328 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:21.017256    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54fc56d44b-zq9ss"
Oct 30 02:41:21.358212 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:21.358126    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.486335 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:21.486297    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:21.624859 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:21.624754    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:24.467540 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:24.467455    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:24.467540 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:24.467513    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:24.472063 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:24.472041    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:25.129061 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:25.129030    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:41:29.424054 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:29.424019    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:29.580031 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:29.579992    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:30.283789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:30.283746    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:30.404192 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:30.404152    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:34.630904 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:41:34.630859    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:41:40.571046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:40.571002    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:40.838230 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:40.838150    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:41.011933 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.011889    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:41.115882 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.115788    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-856b7bc648-r699g" podUID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" containerName="console" containerID="cri-o://be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285" gracePeriod=15
Oct 30 02:41:41.127456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.127425    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:41.397714 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.397692    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:41.476777 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\" (UniqueName: \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.476984 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476791    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.476984 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476847    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.476984 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.476984 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.476984 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476948    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.477230 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.476999    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\") pod \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\" (UID: \"2ad7dc91-5791-4de9-bcdd-e16b537c93df\") "
Oct 30 02:41:41.479919 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.477635    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:41.479919 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.477103    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:41:41.479919 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.477979    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:41.479919 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.478386    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:41:41.480178 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.479935    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:41.482960 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.482933    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "kube-api-access-5zdjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:41:41.483864 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.483846    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config" pod "2ad7dc91-5791-4de9-bcdd-e16b537c93df" (UID: "2ad7dc91-5791-4de9-bcdd-e16b537c93df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:41:41.578021 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.577982    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-oauth-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578021 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578016    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578021 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578030    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578472 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578042    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad7dc91-5791-4de9-bcdd-e16b537c93df-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578472 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578054    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zdjn\" (UniqueName: \"kubernetes.io/projected/2ad7dc91-5791-4de9-bcdd-e16b537c93df-kube-api-access-5zdjn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578472 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578066    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ad7dc91-5791-4de9-bcdd-e16b537c93df-tmp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:41.578472 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:41.578078    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ad7dc91-5791-4de9-bcdd-e16b537c93df-console-oauth-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:41:42.181653 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.181565    4528 generic.go:356] "Generic (PLEG): container finished" podID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" containerID="be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285" exitCode=0
Oct 30 02:41:42.181653 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.181622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856b7bc648-r699g" event={"ID":"2ad7dc91-5791-4de9-bcdd-e16b537c93df","Type":"ContainerDied","Data":"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"}
Oct 30 02:41:42.181653 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.181642    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-856b7bc648-r699g"
Oct 30 02:41:42.181893 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.181667    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-856b7bc648-r699g" event={"ID":"2ad7dc91-5791-4de9-bcdd-e16b537c93df","Type":"ContainerDied","Data":"89d29217c1d051e0a1cca4b90b09b0e786e574900e914b8a434b8b81dbe6f1d5"}
Oct 30 02:41:42.181893 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.181688    4528 scope.go:122] "RemoveContainer" containerID="be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"
Oct 30 02:41:42.189653 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.189631    4528 scope.go:122] "RemoveContainer" containerID="be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"
Oct 30 02:41:42.189956 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:41:42.189925    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285\": container with ID starting with be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285 not found: ID does not exist" containerID="be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"
Oct 30 02:41:42.190023 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.189965    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285"} err="failed to get container status \"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285\": rpc error: code = NotFound desc = could not find container \"be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285\": container with ID starting with be2de230adcc2149dcb1ca1b4e1c6dc4db8b9188f363d310cb5d6801bd3cb285 not found: ID does not exist"
Oct 30 02:41:42.228242 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.228216    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:42.238228 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:42.238206    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-856b7bc648-r699g"]
Oct 30 02:41:43.847085 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:43.847051    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:43.891629 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:43.891588    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" path="/var/lib/kubelet/pods/2ad7dc91-5791-4de9-bcdd-e16b537c93df/volumes"
Oct 30 02:41:43.995680 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:43.995645    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:49.506721 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.506686    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:41:49.507251 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.506951    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" containerName="console"
Oct 30 02:41:49.507251 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.506963    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" containerName="console"
Oct 30 02:41:49.507251 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.507013    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad7dc91-5791-4de9-bcdd-e16b537c93df" containerName="console"
Oct 30 02:41:49.546541 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.546512    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:41:49.546727 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.546638    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.641831 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.641791    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m62j\" (UniqueName: \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642002 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.641850    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642002 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.641957    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642002 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.641986    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642103 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.642074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642136 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.642102    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642136 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.642128    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.642195 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.642162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743109 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743109 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743114    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743315 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743131    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743315 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743153    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m62j\" (UniqueName: \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743315 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743270    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743464 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743337    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743542 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743515    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.743618 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.743592    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.744438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.744372    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.744561 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.744434    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.744561 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.744461    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.744674 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.744590    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.744954 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.744931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.749222 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.748033    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.749222 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.748060    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.751320 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.751292    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:49.764291 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.764239    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m62j\" (UniqueName: \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\") pod \"console-5768fc6bbc-wpv86\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") " pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.870667 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.870633    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:49.893142 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:49.893109    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:50.020301 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:50.020179    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:50.221308 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:50.221263    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:41:50.226661 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:41:50.226632    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f508aa_4909_426d_be6c_5ef1ae42e745.slice/crio-632e048498cf3042a12f3ffaf6e8bcb4648ec91cf9c2a0d46aa002a8970b2c25 WatchSource:0}: Error finding container 632e048498cf3042a12f3ffaf6e8bcb4648ec91cf9c2a0d46aa002a8970b2c25: Status 404 returned error can't find the container with id 632e048498cf3042a12f3ffaf6e8bcb4648ec91cf9c2a0d46aa002a8970b2c25
Oct 30 02:41:51.213110 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:51.213074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5768fc6bbc-wpv86" event={"ID":"50f508aa-4909-426d-be6c-5ef1ae42e745","Type":"ContainerStarted","Data":"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"}
Oct 30 02:41:51.213497 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:51.213124    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5768fc6bbc-wpv86" event={"ID":"50f508aa-4909-426d-be6c-5ef1ae42e745","Type":"ContainerStarted","Data":"632e048498cf3042a12f3ffaf6e8bcb4648ec91cf9c2a0d46aa002a8970b2c25"}
Oct 30 02:41:51.252253 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:51.252199    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5768fc6bbc-wpv86" podStartSLOduration=2.252186313 podStartE2EDuration="2.252186313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:51.251942637 +0000 UTC m=+69.476191913" watchObservedRunningTime="2026-10-30 02:41:51.252186313 +0000 UTC m=+69.476435589"
Oct 30 02:41:58.849245 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:58.849212    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:58.987419 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:58.987356    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:41:59.870880 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:59.870848    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:59.871250 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:59.870932    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:41:59.875450 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:41:59.875426    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:42:00.243686 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:00.243655    4528 generic.go:356] "Generic (PLEG): container finished" podID="43c945bb-127b-4ce1-9807-dc3de3cb297b" containerID="5229218a2781a60c2ff1a647cadf1d85cffdeabfe968371bd9e8870f6398b0f7" exitCode=0
Oct 30 02:42:00.243882 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:00.243729    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g" event={"ID":"43c945bb-127b-4ce1-9807-dc3de3cb297b","Type":"ContainerDied","Data":"5229218a2781a60c2ff1a647cadf1d85cffdeabfe968371bd9e8870f6398b0f7"}
Oct 30 02:42:00.247493 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:00.247468    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:42:00.351409 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:00.351377    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:42:01.396978 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.396950    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:42:01.541123 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541026    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") pod \"43c945bb-127b-4ce1-9807-dc3de3cb297b\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") "
Oct 30 02:42:01.541123 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") pod \"43c945bb-127b-4ce1-9807-dc3de3cb297b\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") "
Oct 30 02:42:01.541123 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541107    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") pod \"43c945bb-127b-4ce1-9807-dc3de3cb297b\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") "
Oct 30 02:42:01.541406 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541131    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") pod \"43c945bb-127b-4ce1-9807-dc3de3cb297b\" (UID: \"43c945bb-127b-4ce1-9807-dc3de3cb297b\") "
Oct 30 02:42:01.541453 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541384    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path" pod "43c945bb-127b-4ce1-9807-dc3de3cb297b" (UID: "43c945bb-127b-4ce1-9807-dc3de3cb297b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:01.541515 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541496    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle" pod "43c945bb-127b-4ce1-9807-dc3de3cb297b" (UID: "43c945bb-127b-4ce1-9807-dc3de3cb297b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:01.541571 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.541552    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle" pod "43c945bb-127b-4ce1-9807-dc3de3cb297b" (UID: "43c945bb-127b-4ce1-9807-dc3de3cb297b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:01.543498 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.543476    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d" pod "43c945bb-127b-4ce1-9807-dc3de3cb297b" (UID: "43c945bb-127b-4ce1-9807-dc3de3cb297b"). InnerVolumeSpecName "kube-api-access-m4n5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:01.641932 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.641891    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4n5d\" (UniqueName: \"kubernetes.io/projected/43c945bb-127b-4ce1-9807-dc3de3cb297b-kube-api-access-m4n5d\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:01.641932 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.641923    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:01.641932 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.641932    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c945bb-127b-4ce1-9807-dc3de3cb297b-service-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:01.641932 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:01.641943    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/43c945bb-127b-4ce1-9807-dc3de3cb297b-archives-path\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:02.250850 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:02.250762    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g"
Oct 30 02:42:02.250850 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:02.250765    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j7nq4-qbr4g" event={"ID":"43c945bb-127b-4ce1-9807-dc3de3cb297b","Type":"ContainerDied","Data":"58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c"}
Oct 30 02:42:02.251026 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:02.250870    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58cfa44f519b05178055f4a6185d62adce4c673dca33a49f175dd3642d753f4c"
Oct 30 02:42:02.727503 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:02.727468    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:02.841215 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:02.841181    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:04.630647 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:42:04.630610    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:42:06.548995 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:06.548958    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:09.376435 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.376401    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-6fc2t-6gzkc"]
Oct 30 02:42:09.376813 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.376656    4528 cpu_manager.go:414] "containerMap: removing container" podUID="43c945bb-127b-4ce1-9807-dc3de3cb297b" containerName="insights-gathering"
Oct 30 02:42:09.376813 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.376666    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c945bb-127b-4ce1-9807-dc3de3cb297b" containerName="insights-gathering"
Oct 30 02:42:09.376813 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.376718    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c945bb-127b-4ce1-9807-dc3de3cb297b" containerName="insights-gathering"
Oct 30 02:42:09.379779 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.379756    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.392407 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.392386    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:42:09.402512 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.402486    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpstg\""
Oct 30 02:42:09.423412 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.423382    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:42:09.424217 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.424198    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6fc2t-6gzkc"]
Oct 30 02:42:09.498339 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.498310    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.498491 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.498363    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.498491 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.498441    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.498573 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.498524    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcps\" (UniqueName: \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599169 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599134    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599189    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599210    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcps\" (UniqueName: \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599655 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.599873 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.599852    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.600043 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.600026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.620996 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.620966    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcps\" (UniqueName: \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\") pod \"periodic-gathering-6fc2t-6gzkc\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") " pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.690739 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.690695    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:42:09.984546 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:09.984468    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-6fc2t-6gzkc"]
Oct 30 02:42:09.989991 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:42:09.989962    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1561e641_f0b5_4427_8b52_9dcddd81cc3d.slice/crio-54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe WatchSource:0}: Error finding container 54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe: Status 404 returned error can't find the container with id 54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe
Oct 30 02:42:10.278669 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:10.278559    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc" event={"ID":"1561e641-f0b5-4427-8b52-9dcddd81cc3d","Type":"ContainerStarted","Data":"b3447a4131e2a33dc25debd135fea243c4fb63e3e89c0cae23864dc5bafc8875"}
Oct 30 02:42:10.278669 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:10.278639    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc" event={"ID":"1561e641-f0b5-4427-8b52-9dcddd81cc3d","Type":"ContainerStarted","Data":"54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe"}
Oct 30 02:42:10.316567 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:10.316492    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc" podStartSLOduration=1.3164794020000001 podStartE2EDuration="1.316479402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:10.316108138 +0000 UTC m=+88.540357416" watchObservedRunningTime="2026-10-30 02:42:10.316479402 +0000 UTC m=+88.540728679"
Oct 30 02:42:11.491578 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.491545    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5f97b844-qh55q"]
Oct 30 02:42:11.494898 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.494875    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.518877 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.518845    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5f97b844-qh55q"]
Oct 30 02:42:11.615124 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615087    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-service-ca\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615124 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615129    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-trusted-ca-bundle\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615163    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c4f1c19-75e8-4985-ac23-5798150dc419-tmp\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615229    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-oauth-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wsk\" (UniqueName: \"kubernetes.io/projected/3c4f1c19-75e8-4985-ac23-5798150dc419-kube-api-access-c5wsk\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615324    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-console-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.615456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.615441    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-oauth-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.669722 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.669690    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:11.716270 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-oauth-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716274    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-service-ca\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716295    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-trusted-ca-bundle\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c4f1c19-75e8-4985-ac23-5798150dc419-tmp\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716359    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-oauth-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716576 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716401    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wsk\" (UniqueName: \"kubernetes.io/projected/3c4f1c19-75e8-4985-ac23-5798150dc419-kube-api-access-c5wsk\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716576 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716451    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-console-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.716576 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.716495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.717140 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.717107    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-oauth-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.717140 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.717128    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c4f1c19-75e8-4985-ac23-5798150dc419-tmp\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.717327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.717128    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-service-ca\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.717327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.717286    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-trusted-ca-bundle\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.717468 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.717444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c4f1c19-75e8-4985-ac23-5798150dc419-console-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.718994 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.718971    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-oauth-config\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.719076 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.719021    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c4f1c19-75e8-4985-ac23-5798150dc419-console-serving-cert\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.738055 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.738030    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wsk\" (UniqueName: \"kubernetes.io/projected/3c4f1c19-75e8-4985-ac23-5798150dc419-kube-api-access-c5wsk\") pod \"console-b5f97b844-qh55q\" (UID: \"3c4f1c19-75e8-4985-ac23-5798150dc419\") " pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.789537 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.789454    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:11.806298 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.806270    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:11.951242 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:11.951205    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:12.102577 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:12.102501    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5f97b844-qh55q"]
Oct 30 02:42:12.107429 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:42:12.107403    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4f1c19_75e8_4985_ac23_5798150dc419.slice/crio-00fff25a84ba2b22f1e006e5b21060f4b69beca2732fd37a9f591f1740181149 WatchSource:0}: Error finding container 00fff25a84ba2b22f1e006e5b21060f4b69beca2732fd37a9f591f1740181149: Status 404 returned error can't find the container with id 00fff25a84ba2b22f1e006e5b21060f4b69beca2732fd37a9f591f1740181149
Oct 30 02:42:12.288050 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:12.288007    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5f97b844-qh55q" event={"ID":"3c4f1c19-75e8-4985-ac23-5798150dc419","Type":"ContainerStarted","Data":"6f6cc927ccb399beaad756195ac60c7bffb75c8025b2638c73ef97b944c35f84"}
Oct 30 02:42:12.288551 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:12.288121    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5f97b844-qh55q" event={"ID":"3c4f1c19-75e8-4985-ac23-5798150dc419","Type":"ContainerStarted","Data":"00fff25a84ba2b22f1e006e5b21060f4b69beca2732fd37a9f591f1740181149"}
Oct 30 02:42:12.328333 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:12.328274    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b5f97b844-qh55q" podStartSLOduration=1.328257137 podStartE2EDuration="1.328257137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:12.327783252 +0000 UTC m=+90.552032529" watchObservedRunningTime="2026-10-30 02:42:12.328257137 +0000 UTC m=+90.552506415"
Oct 30 02:42:19.169699 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:19.169658    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:19.301913 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:19.301879    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:19.772036 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:19.771994    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:21.807161 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:21.807124    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:21.807571 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:21.807189    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:21.812068 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:21.812042    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:22.325484 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:22.325458    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5f97b844-qh55q"
Oct 30 02:42:22.396828 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:22.396783    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:42:25.370357 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.370299    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c9ccb594d-v6l8z" podUID="7a166b83-a7d0-4fca-b729-e6a989558098" containerName="console" containerID="cri-o://693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b" gracePeriod=15
Oct 30 02:42:25.627117 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.627061    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:42:25.722417 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722377    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722559 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722437    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722559 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722535    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722656 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722656 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722602    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\" (UniqueName: \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722752 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722658    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722752 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722690    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722752 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722722    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\") pod \"7a166b83-a7d0-4fca-b729-e6a989558098\" (UID: \"7a166b83-a7d0-4fca-b729-e6a989558098\") "
Oct 30 02:42:25.722937 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722804    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:25.722937 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722871    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:25.723070 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.722941    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:25.723139 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.723127    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a166b83-a7d0-4fca-b729-e6a989558098-tmp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.723188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.723143    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.723188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.723164    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.723188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.723173    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:25.723390 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.723166    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:25.724789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.724764    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "kube-api-access-rxxv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:25.724789 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.724776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:25.724955 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.724856    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config" pod "7a166b83-a7d0-4fca-b729-e6a989558098" (UID: "7a166b83-a7d0-4fca-b729-e6a989558098"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:25.824046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.824010    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.824046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.824039    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxv7\" (UniqueName: \"kubernetes.io/projected/7a166b83-a7d0-4fca-b729-e6a989558098-kube-api-access-rxxv7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.824046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.824052    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-console-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.824278 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.824061    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a166b83-a7d0-4fca-b729-e6a989558098-oauth-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:25.824278 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:25.824070    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a166b83-a7d0-4fca-b729-e6a989558098-console-oauth-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:26.336069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.335981    4528 generic.go:356] "Generic (PLEG): container finished" podID="7a166b83-a7d0-4fca-b729-e6a989558098" containerID="693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b" exitCode=0
Oct 30 02:42:26.336069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.336029    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9ccb594d-v6l8z"
Oct 30 02:42:26.336236 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.336067    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9ccb594d-v6l8z" event={"ID":"7a166b83-a7d0-4fca-b729-e6a989558098","Type":"ContainerDied","Data":"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"}
Oct 30 02:42:26.336236 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.336116    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9ccb594d-v6l8z" event={"ID":"7a166b83-a7d0-4fca-b729-e6a989558098","Type":"ContainerDied","Data":"a57d76c939b07097cc92f6c6dc5f2d77e960322936b6e5de5cc79a0dae1cfbd4"}
Oct 30 02:42:26.336236 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.336136    4528 scope.go:122] "RemoveContainer" containerID="693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"
Oct 30 02:42:26.343620 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.343599    4528 scope.go:122] "RemoveContainer" containerID="693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"
Oct 30 02:42:26.343909 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:42:26.343880    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b\": container with ID starting with 693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b not found: ID does not exist" containerID="693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"
Oct 30 02:42:26.344011 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.343908    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b"} err="failed to get container status \"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b\": rpc error: code = NotFound desc = could not find container \"693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b\": container with ID starting with 693b93e73cdf6dbcd269dbf6bacd9a173aee741e9eeb26109f78c9250113673b not found: ID does not exist"
Oct 30 02:42:26.409731 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.409706    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:42:26.419796 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:26.419772    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9ccb594d-v6l8z"]
Oct 30 02:42:27.892009 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:27.891977    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a166b83-a7d0-4fca-b729-e6a989558098" path="/var/lib/kubelet/pods/7a166b83-a7d0-4fca-b729-e6a989558098/volumes"
Oct 30 02:42:28.892267 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:28.892229    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:29.010268 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:29.010223    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:29.139249 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:29.139214    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:29.537842 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:29.537787    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:42:34.630224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:42:34.630181    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:42:47.416600 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.416543    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5768fc6bbc-wpv86" podUID="50f508aa-4909-426d-be6c-5ef1ae42e745" containerName="console" containerID="cri-o://86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6" gracePeriod=15
Oct 30 02:42:47.675672 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.675612    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:42:47.799323 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799280    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799323 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799326    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799388    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799406    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799428    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799457    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\" (UniqueName: \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\") pod \"50f508aa-4909-426d-be6c-5ef1ae42e745\" (UID: \"50f508aa-4909-426d-be6c-5ef1ae42e745\") "
Oct 30 02:42:47.799926 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799893    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:42:47.799981 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.799955    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:47.800068 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.800032    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:47.800130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.800093    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:47.800200 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.800175    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:42:47.801624 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.801597    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:47.801728 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.801627    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:42:47.801728 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.801648    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j" pod "50f508aa-4909-426d-be6c-5ef1ae42e745" (UID: "50f508aa-4909-426d-be6c-5ef1ae42e745"). InnerVolumeSpecName "kube-api-access-2m62j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:42:47.900308 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900283    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m62j\" (UniqueName: \"kubernetes.io/projected/50f508aa-4909-426d-be6c-5ef1ae42e745-kube-api-access-2m62j\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900308 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900309    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-oauth-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900319    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900329    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50f508aa-4909-426d-be6c-5ef1ae42e745-console-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900337    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-service-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900346    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-console-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900353    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50f508aa-4909-426d-be6c-5ef1ae42e745-oauth-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:47.900438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:47.900361    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f508aa-4909-426d-be6c-5ef1ae42e745-tmp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:42:48.410170 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.410130    4528 generic.go:356] "Generic (PLEG): container finished" podID="50f508aa-4909-426d-be6c-5ef1ae42e745" containerID="86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6" exitCode=0
Oct 30 02:42:48.410372 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.410185    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5768fc6bbc-wpv86"
Oct 30 02:42:48.410372 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.410225    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5768fc6bbc-wpv86" event={"ID":"50f508aa-4909-426d-be6c-5ef1ae42e745","Type":"ContainerDied","Data":"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"}
Oct 30 02:42:48.410372 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.410290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5768fc6bbc-wpv86" event={"ID":"50f508aa-4909-426d-be6c-5ef1ae42e745","Type":"ContainerDied","Data":"632e048498cf3042a12f3ffaf6e8bcb4648ec91cf9c2a0d46aa002a8970b2c25"}
Oct 30 02:42:48.410372 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.410319    4528 scope.go:122] "RemoveContainer" containerID="86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"
Oct 30 02:42:48.418338 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.418173    4528 scope.go:122] "RemoveContainer" containerID="86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"
Oct 30 02:42:48.418585 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:42:48.418442    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6\": container with ID starting with 86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6 not found: ID does not exist" containerID="86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"
Oct 30 02:42:48.418585 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.418464    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6"} err="failed to get container status \"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6\": rpc error: code = NotFound desc = could not find container \"86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6\": container with ID starting with 86bd2924c24a01ed6b0ba304641fa0c4d80dfdb08e38c03ac17a4eafa42883c6 not found: ID does not exist"
Oct 30 02:42:48.454680 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.454651    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:42:48.464787 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:48.464759    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5768fc6bbc-wpv86"]
Oct 30 02:42:49.890790 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:42:49.890759    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f508aa-4909-426d-be6c-5ef1ae42e745" path="/var/lib/kubelet/pods/50f508aa-4909-426d-be6c-5ef1ae42e745/volumes"
Oct 30 02:43:04.630500 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:43:04.630420    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:43:29.692987 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:29.692951    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:43:34.562358 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:34.562324    4528 generic.go:356] "Generic (PLEG): container finished" podID="1561e641-f0b5-4427-8b52-9dcddd81cc3d" containerID="b3447a4131e2a33dc25debd135fea243c4fb63e3e89c0cae23864dc5bafc8875" exitCode=0
Oct 30 02:43:34.562738 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:34.562363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc" event={"ID":"1561e641-f0b5-4427-8b52-9dcddd81cc3d","Type":"ContainerDied","Data":"b3447a4131e2a33dc25debd135fea243c4fb63e3e89c0cae23864dc5bafc8875"}
Oct 30 02:43:34.630722 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:43:34.630687    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:43:35.688415 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.688386    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:43:35.756923 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.756890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\") pod \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") "
Oct 30 02:43:35.757101 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.756930    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\" (UniqueName: \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\") pod \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") "
Oct 30 02:43:35.757101 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.756960    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\") pod \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") "
Oct 30 02:43:35.757101 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.756996    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\") pod \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\" (UID: \"1561e641-f0b5-4427-8b52-9dcddd81cc3d\") "
Oct 30 02:43:35.757345 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.757314    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path" pod "1561e641-f0b5-4427-8b52-9dcddd81cc3d" (UID: "1561e641-f0b5-4427-8b52-9dcddd81cc3d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:43:35.757407 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.757327    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle" pod "1561e641-f0b5-4427-8b52-9dcddd81cc3d" (UID: "1561e641-f0b5-4427-8b52-9dcddd81cc3d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:35.757407 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.757362    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle" pod "1561e641-f0b5-4427-8b52-9dcddd81cc3d" (UID: "1561e641-f0b5-4427-8b52-9dcddd81cc3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:43:35.759028 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.759005    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps" pod "1561e641-f0b5-4427-8b52-9dcddd81cc3d" (UID: "1561e641-f0b5-4427-8b52-9dcddd81cc3d"). InnerVolumeSpecName "kube-api-access-2tcps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:43:35.858455 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.858382    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1561e641-f0b5-4427-8b52-9dcddd81cc3d-archives-path\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:43:35.858455 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.858408    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcps\" (UniqueName: \"kubernetes.io/projected/1561e641-f0b5-4427-8b52-9dcddd81cc3d-kube-api-access-2tcps\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:43:35.858455 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.858422    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-service-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:43:35.858455 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:35.858430    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1561e641-f0b5-4427-8b52-9dcddd81cc3d-trusted-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:43:36.569760 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:36.569731    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc"
Oct 30 02:43:36.569938 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:36.569730    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-6fc2t-6gzkc" event={"ID":"1561e641-f0b5-4427-8b52-9dcddd81cc3d","Type":"ContainerDied","Data":"54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe"}
Oct 30 02:43:36.569938 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:43:36.569864    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54907d0ad05987ba4fb006a7103f95275b1791ac53371a457d50e39f24eb41fe"
Oct 30 02:44:04.630547 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:44:04.630511    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:44:09.436827 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:44:09.436754    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:09.562680 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:44:09.562639    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:27.046602 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:44:27.046518    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:27.170715 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:44:27.170686    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:29.865455 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:44:29.865418    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:44:34.630979 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:44:34.630935    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:45:04.630501 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:45:04.630459    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:45:29.944960 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:45:29.944921    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:45:34.630380 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:45:34.630339    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:46:04.631118 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:46:04.631036    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:46:30.076831 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:46:30.076786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:46:34.630996 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:46:34.630957    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:47:04.630604 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:47:04.630566    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:47:13.417734 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:47:13.417699    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:30.184377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:47:30.184288    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:47:34.631599 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:47:34.631560    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:47:41.939604 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:47:41.939565    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:04.630259 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:48:04.630218    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:48:30.344217 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:48:30.344179    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:48:34.628795 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:48:34.628764    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:48:52.059031 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:48:52.058990    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:04.630887 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:49:04.630792    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:49:30.492724 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:49:30.492686    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:49:34.630779 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:49:34.630742    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:50:04.631946 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:50:04.631841    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:50:04.848109 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:50:04.848078    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:30.640945 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:50:30.640858    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:50:34.629847 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:50:34.629791    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:51:04.630597 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:51:04.630560    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:51:30.748654 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:51:30.748615    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:51:34.630988 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:51:34.630950    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:52:04.632694 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:52:04.632608    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:52:30.858489 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:52:30.858441    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:52:34.630495 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:52:34.630459    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:53:04.630445 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:53:04.630405    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:53:12.995730 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:12.995693    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:13.135024 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:13.134991    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:26.514774 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:26.514693    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:30.974065 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:30.974024    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:34.631310 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:53:34.631267    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:53:35.036928 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:35.036843    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:35.840666 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:35.840624    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:38.480771 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:38.480733    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:38.651512 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:38.651480    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:41.336284 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.336249    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:41.490062 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.490020    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:41.691215 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691182    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"]
Oct 30 02:53:41.691464 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691447    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7a166b83-a7d0-4fca-b729-e6a989558098" containerName="console"
Oct 30 02:53:41.691520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691465    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a166b83-a7d0-4fca-b729-e6a989558098" containerName="console"
Oct 30 02:53:41.691520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691492    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50f508aa-4909-426d-be6c-5ef1ae42e745" containerName="console"
Oct 30 02:53:41.691520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691497    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f508aa-4909-426d-be6c-5ef1ae42e745" containerName="console"
Oct 30 02:53:41.691520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691506    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1561e641-f0b5-4427-8b52-9dcddd81cc3d" containerName="insights-gathering"
Oct 30 02:53:41.691520 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691513    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1561e641-f0b5-4427-8b52-9dcddd81cc3d" containerName="insights-gathering"
Oct 30 02:53:41.691692 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691562    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f508aa-4909-426d-be6c-5ef1ae42e745" containerName="console"
Oct 30 02:53:41.691692 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691572    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1561e641-f0b5-4427-8b52-9dcddd81cc3d" containerName="insights-gathering"
Oct 30 02:53:41.691692 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.691579    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a166b83-a7d0-4fca-b729-e6a989558098" containerName="console"
Oct 30 02:53:41.694456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.694435    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.717122 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.717096    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"]
Oct 30 02:53:41.726238 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.726218    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 02:53:41.726332 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.726318    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 02:53:41.726874 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.726854    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g7qzx\""
Oct 30 02:53:41.762806 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.762775    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk7n\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-kube-api-access-kqk7n\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.762943 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.762847    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.863896 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.863866    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.864028 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.863909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk7n\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-kube-api-access-kqk7n\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.897758 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.897731    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:41.908259 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.908239    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 02:53:41.928169 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.928147    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 02:53:41.939902 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:41.939880    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk7n\" (UniqueName: \"kubernetes.io/projected/d9c4a516-58c1-4cab-92d4-652288e2828d-kube-api-access-kqk7n\") pod \"cert-manager-webhook-5dbbbf75cc-fqvwn\" (UID: \"d9c4a516-58c1-4cab-92d4-652288e2828d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:42.038224 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:42.038155    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g7qzx\""
Oct 30 02:53:42.046362 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:42.046342    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:42.344127 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:42.344049    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"]
Oct 30 02:53:42.348861 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:53:42.348789    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c4a516_58c1_4cab_92d4_652288e2828d.slice/crio-5adeb29cdb8fcc896a4142ea935aa52b442bd64f39638f0161520b24222aebdb WatchSource:0}: Error finding container 5adeb29cdb8fcc896a4142ea935aa52b442bd64f39638f0161520b24222aebdb: Status 404 returned error can't find the container with id 5adeb29cdb8fcc896a4142ea935aa52b442bd64f39638f0161520b24222aebdb
Oct 30 02:53:42.350799 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:42.350783    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:53:42.509374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:42.509339    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn" event={"ID":"d9c4a516-58c1-4cab-92d4-652288e2828d","Type":"ContainerStarted","Data":"5adeb29cdb8fcc896a4142ea935aa52b442bd64f39638f0161520b24222aebdb"}
Oct 30 02:53:43.084258 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:43.084223    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:43.241626 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:43.241592    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:43.759449 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:43.759408    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:44.057690 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:44.052306    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:46.524369 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:46.524335    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn" event={"ID":"d9c4a516-58c1-4cab-92d4-652288e2828d","Type":"ContainerStarted","Data":"a69bf170e24dcefbaae24447cc3bfc2486c135089ca2173fe767fab9eaf52302"}
Oct 30 02:53:46.524809 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:46.524545    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:46.561084 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:46.561032    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn" podStartSLOduration=2.326760912 podStartE2EDuration="5.561019496s" totalImagesPullingTime="3.234258584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:46.560272517 +0000 UTC m=+784.784521804" watchObservedRunningTime="2026-10-30 02:53:46.561019496 +0000 UTC m=+784.785268774"
Oct 30 02:53:51.008978 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:51.008934    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:53:52.531011 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:52.530973    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fqvwn"
Oct 30 02:53:53.215970 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.215934    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tnssn"]
Oct 30 02:53:53.219326 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.219302    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.243152 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.243127    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sf68w\""
Oct 30 02:53:53.244413 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.244390    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tnssn"]
Oct 30 02:53:53.345641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.345609    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-bound-sa-token\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.345641 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.345650    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjs6\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-kube-api-access-bgjs6\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.446132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.446100    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-bound-sa-token\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.446132 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.446144    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjs6\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-kube-api-access-bgjs6\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.469521 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.469443    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-bound-sa-token\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.474415 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.474389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjs6\" (UniqueName: \"kubernetes.io/projected/336dd144-1d7a-4e93-9670-8cd54c3f4e9c-kube-api-access-bgjs6\") pod \"cert-manager-597b95cb84-tnssn\" (UID: \"336dd144-1d7a-4e93-9670-8cd54c3f4e9c\") " pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.529687 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.529653    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tnssn"
Oct 30 02:53:53.823554 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:53.823467    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tnssn"]
Oct 30 02:53:53.828780 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:53:53.828748    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod336dd144_1d7a_4e93_9670_8cd54c3f4e9c.slice/crio-7af9edb20d708fefed7fd0aea33643d3db534b0e3057530e5b66f86dce47f05d WatchSource:0}: Error finding container 7af9edb20d708fefed7fd0aea33643d3db534b0e3057530e5b66f86dce47f05d: Status 404 returned error can't find the container with id 7af9edb20d708fefed7fd0aea33643d3db534b0e3057530e5b66f86dce47f05d
Oct 30 02:53:54.553012 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:54.552976    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tnssn" event={"ID":"336dd144-1d7a-4e93-9670-8cd54c3f4e9c","Type":"ContainerStarted","Data":"c5fba4c27e290d83c353a190d3a52e43152daef9a86a1909dc81b0c6e71346bb"}
Oct 30 02:53:54.553187 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:54.553021    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tnssn" event={"ID":"336dd144-1d7a-4e93-9670-8cd54c3f4e9c","Type":"ContainerStarted","Data":"7af9edb20d708fefed7fd0aea33643d3db534b0e3057530e5b66f86dce47f05d"}
Oct 30 02:53:54.593192 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:53:54.593146    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tnssn" podStartSLOduration=1.593131794 podStartE2EDuration="1.593131794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:53:54.59209045 +0000 UTC m=+792.816339706" watchObservedRunningTime="2026-10-30 02:53:54.593131794 +0000 UTC m=+792.817381070"
Oct 30 02:54:04.630925 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:54:04.630890    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:54:19.426302 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:19.426227    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:19.557673 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:19.557642    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:28.593880 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:28.593840    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:28.752249 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:28.752205    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:28.940987 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:28.940954    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:29.082579 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:29.082541    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:29.341268 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:29.341153    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:31.100077 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:54:31.100040    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:34.630883 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:54:34.630848    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:55:04.630599 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:55:04.630558    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:55:31.311554 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:55:31.311513    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:34.630510 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:55:34.630476    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:55:52.808675 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:55:52.808640    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:55:52.955394 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:55:52.955361    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.240119 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:02.240077    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.401297 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:02.401261    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.820032 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:02.819995    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:02.955661 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:02.955630    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:04.631072 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:56:04.631037    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:56:05.339436 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:05.339396    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:05.501128 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:05.501089    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:06.511595 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:06.511554    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:07.492442 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:07.492409    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:07.664683 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:07.664646    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:07.843875 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:07.843782    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.042524 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.042487    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.187345 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.187313    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.350595 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.350565    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"]
Oct 30 02:56:08.351684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.351654    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.354051 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.354036    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.363321 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.363300    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.363456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.363326    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7m6j\" (UniqueName: \"kubernetes.io/projected/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-kube-api-access-w7m6j\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.363456 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.363362    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.381300 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.381282    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:56:08.381300 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.381281    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\""
Oct 30 02:56:08.407069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.407042    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"]
Oct 30 02:56:08.439865 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.439779    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\""
Oct 30 02:56:08.439988 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.439903    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-hdzf5\""
Oct 30 02:56:08.464304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.464275    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.464419 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.464324    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.464419 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.464341    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7m6j\" (UniqueName: \"kubernetes.io/projected/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-kube-api-access-w7m6j\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.466675 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.466650    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-apiservice-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.466791 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.466771    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-webhook-cert\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.486181 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.486159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7m6j\" (UniqueName: \"kubernetes.io/projected/3a22d0a0-6c60-46b0-ae2c-019e913e4ece-kube-api-access-w7m6j\") pod \"opentelemetry-operator-webhook-dd54bd85d-22cg4\" (UID: \"3a22d0a0-6c60-46b0-ae2c-019e913e4ece\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:08.496327 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.496303    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:08.665075 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:08.665048    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:09.003569 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:09.003489    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"]
Oct 30 02:56:09.020670 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:56:09.020637    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a22d0a0_6c60_46b0_ae2c_019e913e4ece.slice/crio-98a0e18865b96ea1d03815ffb62c5784add684fe2845bc519ab83bcd880edb6e WatchSource:0}: Error finding container 98a0e18865b96ea1d03815ffb62c5784add684fe2845bc519ab83bcd880edb6e: Status 404 returned error can't find the container with id 98a0e18865b96ea1d03815ffb62c5784add684fe2845bc519ab83bcd880edb6e
Oct 30 02:56:09.036363 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:09.036320    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:09.670997 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:09.670966    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:09.916499 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:09.916463    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:10.001678 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:10.001595    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4" event={"ID":"3a22d0a0-6c60-46b0-ae2c-019e913e4ece","Type":"ContainerStarted","Data":"98a0e18865b96ea1d03815ffb62c5784add684fe2845bc519ab83bcd880edb6e"}
Oct 30 02:56:10.160266 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:10.160220    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:10.809698 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:10.809652    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:11.007374 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:11.007333    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4" event={"ID":"3a22d0a0-6c60-46b0-ae2c-019e913e4ece","Type":"ContainerStarted","Data":"10653b7a70c5700f06d5c2b95b7e7f54b84351a5d5510d4786cbb5045e52ff16"}
Oct 30 02:56:11.007557 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:11.007447    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:11.068896 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:11.068838    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4" podStartSLOduration=1.467907233 podStartE2EDuration="3.068810174s" totalImagesPullingTime="1.600902941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:11.06835227 +0000 UTC m=+929.292601546" watchObservedRunningTime="2026-10-30 02:56:11.068810174 +0000 UTC m=+929.293059453"
Oct 30 02:56:11.219176 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:11.219139    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:11.404808 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:11.404768    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.098958 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:12.098923    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.362801 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:12.362718    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.581580 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:12.581543    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:12.780388 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:12.780357    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:13.475796 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:13.475761    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:13.986687 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:13.986642    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.156289 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:14.156245    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:14.324025 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:14.323943    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:15.670161 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:15.670130    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:15.833646 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:15.833609    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:16.814528 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:16.814492    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.021135 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.021092    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.213290 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.213258    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.317629 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.317597    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"]
Oct 30 02:56:17.321186 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.321164    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"
Oct 30 02:56:17.352654 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.352627    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"]
Oct 30 02:56:17.359303 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.359280    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\""
Oct 30 02:56:17.359538 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.359519    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-dockercfg-jmbhm\""
Oct 30 02:56:17.359589 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.359533    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:56:17.364745 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.364720    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.436699 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.436669    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhr98\" (UniqueName: \"kubernetes.io/projected/e7279edf-52f9-4fed-9271-57adafa3e9ce-kube-api-access-xhr98\") pod \"obo-prometheus-operator-6ddb664c4f-cjmkb\" (UID: \"e7279edf-52f9-4fed-9271-57adafa3e9ce\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"
Oct 30 02:56:17.537085 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.537004    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhr98\" (UniqueName: \"kubernetes.io/projected/e7279edf-52f9-4fed-9271-57adafa3e9ce-kube-api-access-xhr98\") pod \"obo-prometheus-operator-6ddb664c4f-cjmkb\" (UID: \"e7279edf-52f9-4fed-9271-57adafa3e9ce\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"
Oct 30 02:56:17.557013 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.556972    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.560798 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.560777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhr98\" (UniqueName: \"kubernetes.io/projected/e7279edf-52f9-4fed-9271-57adafa3e9ce-kube-api-access-xhr98\") pod \"obo-prometheus-operator-6ddb664c4f-cjmkb\" (UID: \"e7279edf-52f9-4fed-9271-57adafa3e9ce\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"
Oct 30 02:56:17.633273 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.633242    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"
Oct 30 02:56:17.734270 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.734236    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:17.964341 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:17.964310    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb"]
Oct 30 02:56:17.965190 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:56:17.965162    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7279edf_52f9_4fed_9271_57adafa3e9ce.slice/crio-e52c904b20b6e14b48b86e038393f0f6188947da1bba1291ebfb7fd49e02b23e WatchSource:0}: Error finding container e52c904b20b6e14b48b86e038393f0f6188947da1bba1291ebfb7fd49e02b23e: Status 404 returned error can't find the container with id e52c904b20b6e14b48b86e038393f0f6188947da1bba1291ebfb7fd49e02b23e
Oct 30 02:56:18.000813 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:18.000779    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:18.032834 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:18.032796    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb" event={"ID":"e7279edf-52f9-4fed-9271-57adafa3e9ce","Type":"ContainerStarted","Data":"e52c904b20b6e14b48b86e038393f0f6188947da1bba1291ebfb7fd49e02b23e"}
Oct 30 02:56:18.303052 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:18.302966    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:18.456293 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:18.456256    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:19.863985 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:19.863944    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:20.644463 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:20.644402    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:20.923451 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:20.923358    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:21.047719 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:21.047680    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb" event={"ID":"e7279edf-52f9-4fed-9271-57adafa3e9ce","Type":"ContainerStarted","Data":"127bf97450b72987741abebd73bc7b616905a49b89c0e5a46506920e096976ab"}
Oct 30 02:56:22.015165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:22.015136    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-dd54bd85d-22cg4"
Oct 30 02:56:22.089388 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:22.089335    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-cjmkb" podStartSLOduration=2.835626763 podStartE2EDuration="5.089321558s" totalImagesPullingTime="2.253694795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:21.090330021 +0000 UTC m=+939.314579297" watchObservedRunningTime="2026-10-30 02:56:22.089321558 +0000 UTC m=+940.313570837"
Oct 30 02:56:30.550265 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:30.550170    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:30.710537 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:30.710499    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:30.864601 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:30.864507    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:31.000900 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:31.000853    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:31.381366 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:31.381331    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:31.515360 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:31.515327    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:34.630782 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:56:34.630744    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:56:46.009935 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:46.009895    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:46.151596 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:46.151563    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:54.563186 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:54.563148    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:54.705145 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:54.705109    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:55.036549 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:55.036513    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:55.197387 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:55.197336    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:56.495755 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:56.495702    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:57.014675 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:57.014634    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:57.200698 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:57.200661    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:58.519635 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.519597    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:58.690115 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.690082    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"]
Oct 30 02:56:58.695949 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.695927    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.704666 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.704635    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:58.713403 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.713386    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-controller-manager-service-cert\""
Oct 30 02:56:58.721186 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.721166    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-controller-manager-dockercfg-kxg9m\""
Oct 30 02:56:58.721336 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.721190    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-metrics\""
Oct 30 02:56:58.721336 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.721260    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"openshift-service-ca.crt\""
Oct 30 02:56:58.721494 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.721314    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"loki-operator-manager-config\""
Oct 30 02:56:58.721547 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.721523    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators-redhat\"/\"kube-root-ca.crt\""
Oct 30 02:56:58.722441 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.722422    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"]
Oct 30 02:56:58.857156 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.857075    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.857156 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.857108    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2jt\" (UniqueName: \"kubernetes.io/projected/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-kube-api-access-cp2jt\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.857156 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.857137    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-manager-config\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.857376 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.857200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-webhook-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.857376 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.857327    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-apiservice-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.958550 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.958513    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.958550 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.958555    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2jt\" (UniqueName: \"kubernetes.io/projected/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-kube-api-access-cp2jt\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.958763 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.958599    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-manager-config\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.958763 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.958624    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-webhook-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.958763 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.958666    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-apiservice-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.959431 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.959398    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-manager-config\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.961130 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.961112    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"loki-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-loki-operator-metrics-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.961294 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.961273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-webhook-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.961337 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.961273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-apiservice-cert\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:58.982666 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:58.982635    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2jt\" (UniqueName: \"kubernetes.io/projected/58fa468d-4152-4a8e-ac79-e515e9ee2e0a-kube-api-access-cp2jt\") pod \"loki-operator-controller-manager-c4d497f54-2dt62\" (UID: \"58fa468d-4152-4a8e-ac79-e515e9ee2e0a\") " pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:59.007267 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.007237    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:56:59.221543 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.221508    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.337213 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.337178    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"]
Oct 30 02:56:59.372566 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.372523    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.642538 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.642502    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:56:59.814659 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:56:59.814615    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:00.191874 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:00.191840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62" event={"ID":"58fa468d-4152-4a8e-ac79-e515e9ee2e0a","Type":"ContainerStarted","Data":"0e9dd04b27d7498046955c97b0372e71d0612bafb64c12068a9a066d3f5f3d2c"}
Oct 30 02:57:01.716192 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:01.716149    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:01.856740 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:01.855413    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:03.206936 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:03.206900    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62" event={"ID":"58fa468d-4152-4a8e-ac79-e515e9ee2e0a","Type":"ContainerStarted","Data":"a97308b8d5ce62c9421412724f5bc76883309d2592fa5bfbca99409a66bf74c4"}
Oct 30 02:57:03.207425 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:03.207027    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:57:03.267653 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:03.267603    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62" podStartSLOduration=2.084422707 podStartE2EDuration="5.267588807s" totalImagesPullingTime="3.1831661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:03.257471804 +0000 UTC m=+981.481721081" watchObservedRunningTime="2026-10-30 02:57:03.267588807 +0000 UTC m=+981.491838083"
Oct 30 02:57:04.631006 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:04.630964    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:57:14.213157 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:14.213124    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators-redhat/loki-operator-controller-manager-c4d497f54-2dt62"
Oct 30 02:57:17.273314 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:17.273279    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:17.406486 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:17.406450    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:22.649788 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:22.649749    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:22.787651 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:22.787613    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:23.604990 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.604956    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:23.671737 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.671695    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:23.675435 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.675414    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.704331 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.704302    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:23.710006 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.709984    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:57:23.710006 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.709982    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:57:23.710165 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.710040    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 02:57:23.712742 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.712721    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 02:57:23.712742 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.712723    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qm2pj\""
Oct 30 02:57:23.746251 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.746211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.746445 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.746260    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.746445 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.746397    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlq7v\" (UniqueName: \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.756438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.756406    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:23.847419 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.847381    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlq7v\" (UniqueName: \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.847664 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.847441    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.847664 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.847476    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.848192 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.848171    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.849894 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.849872    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.868981 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.868916    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlq7v\" (UniqueName: \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\") pod \"data-science-collector-collector-1\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:23.928304 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.928256    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:23.990054 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:23.990020    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:24.284776 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:24.284741    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:24.289956 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:57:24.289926    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2e10b8_dd48_4a96_9715_63b3cc6874db.slice/crio-de9ff876372c9c2763b85c5af8ea2d4f4d4b38eabe2705add9f7ab8837d6cb1b WatchSource:0}: Error finding container de9ff876372c9c2763b85c5af8ea2d4f4d4b38eabe2705add9f7ab8837d6cb1b: Status 404 returned error can't find the container with id de9ff876372c9c2763b85c5af8ea2d4f4d4b38eabe2705add9f7ab8837d6cb1b
Oct 30 02:57:25.121094 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:25.121056    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:25.255311 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:25.255275    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:25.303044 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:25.302998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eb2e10b8-dd48-4a96-9715-63b3cc6874db","Type":"ContainerStarted","Data":"de9ff876372c9c2763b85c5af8ea2d4f4d4b38eabe2705add9f7ab8837d6cb1b"}
Oct 30 02:57:25.383179 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:25.383138    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:26.142560 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:26.142517    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:26.274918 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:26.274881    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:26.753278 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:26.753246    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:27.439438 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:27.439399    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:27.582880 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:27.582843    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:28.227097 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.227061    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:28.230375 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.230351    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.242787 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.242765    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:28.242857 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.242813    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:28.243378 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.243357    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 02:57:28.243451 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.243360    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 02:57:28.253168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253146    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 02:57:28.253168 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253156    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 02:57:28.253319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253149    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 02:57:28.253319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253192    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 02:57:28.253319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253231    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 02:57:28.253319 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.253290    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-nfv94\""
Oct 30 02:57:28.256276 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.256223    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:28.318886 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.318861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eb2e10b8-dd48-4a96-9715-63b3cc6874db","Type":"ContainerStarted","Data":"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"}
Oct 30 02:57:28.356360 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.356316    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.406196078 podStartE2EDuration="5.35630494s" totalImagesPullingTime="3.950108862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:28.356187326 +0000 UTC m=+1006.580436604" watchObservedRunningTime="2026-10-30 02:57:28.35630494 +0000 UTC m=+1006.580554216"
Oct 30 02:57:28.384529 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384491    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384679 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384548    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384679 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384611    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqqs\" (UniqueName: \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384679 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384654    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384800 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384679    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384854 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.384943 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.384916    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.385031 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.385010    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.428363 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.428332    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:28.486252 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486176    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486252 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486235    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486266    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486349    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486386    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqqs\" (UniqueName: \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486446    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486474    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.486764 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.486526    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.487123 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.487067    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.487231 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.487206    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.487388 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.487370    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.487753 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.487727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.488943 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.488919    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.489038 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.488940    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.489038 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.488946    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:28.508929 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:28.508899    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqqs\" (UniqueName: \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:31.513280 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:31.513239    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:32.518133 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.518099    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.543134 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.543098    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.619431 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.619389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.619682 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.619532    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.631834 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.631794    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.812048 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.812013    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:32.833540 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:32.833515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:33.046997 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:33.046955    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:33.378588 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:33.378560    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:33.382696 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:57:33.382668    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816655ab_6ab1_41b0_a303_de76659b0ad9.slice/crio-39b690546a0929845e924f576b922e69e65a2fe7bdb33cce6933e103ced66e07 WatchSource:0}: Error finding container 39b690546a0929845e924f576b922e69e65a2fe7bdb33cce6933e103ced66e07: Status 404 returned error can't find the container with id 39b690546a0929845e924f576b922e69e65a2fe7bdb33cce6933e103ced66e07
Oct 30 02:57:34.345135 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:34.345094    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerStarted","Data":"39b690546a0929845e924f576b922e69e65a2fe7bdb33cce6933e103ced66e07"}
Oct 30 02:57:34.634633 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:34.634595    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:57:36.355191 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:36.355155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerStarted","Data":"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"}
Oct 30 02:57:38.365377 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:38.365287    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerStarted","Data":"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"}
Oct 30 02:57:41.011196 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.011162    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:41.012859 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.012800    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" containerName="otc-container" containerID="cri-o://824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96" gracePeriod=30
Oct 30 02:57:41.084113 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.084080    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:41.270971 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.270915    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:41.281898 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.281877    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\" (UniqueName: \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\") pod \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") "
Oct 30 02:57:41.281989 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.281976    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\") pod \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") "
Oct 30 02:57:41.282036 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.281998    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\" (UniqueName: \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\") pod \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\" (UID: \"eb2e10b8-dd48-4a96-9715-63b3cc6874db\") "
Oct 30 02:57:41.282388 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.282362    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal" pod "eb2e10b8-dd48-4a96-9715-63b3cc6874db" (UID: "eb2e10b8-dd48-4a96-9715-63b3cc6874db"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:41.284146 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.284114    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v" pod "eb2e10b8-dd48-4a96-9715-63b3cc6874db" (UID: "eb2e10b8-dd48-4a96-9715-63b3cc6874db"). InnerVolumeSpecName "kube-api-access-qlq7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:41.284329 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.284311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs" pod "eb2e10b8-dd48-4a96-9715-63b3cc6874db" (UID: "eb2e10b8-dd48-4a96-9715-63b3cc6874db"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:41.378234 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.378200    4528 generic.go:356] "Generic (PLEG): container finished" podID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" containerID="824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96" exitCode=0
Oct 30 02:57:41.378405 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.378251    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:41.378405 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.378294    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eb2e10b8-dd48-4a96-9715-63b3cc6874db","Type":"ContainerDied","Data":"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"}
Oct 30 02:57:41.378405 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.378346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eb2e10b8-dd48-4a96-9715-63b3cc6874db","Type":"ContainerDied","Data":"de9ff876372c9c2763b85c5af8ea2d4f4d4b38eabe2705add9f7ab8837d6cb1b"}
Oct 30 02:57:41.378405 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.378375    4528 scope.go:122] "RemoveContainer" containerID="824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"
Oct 30 02:57:41.380661 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.380625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerStarted","Data":"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"}
Oct 30 02:57:41.380870 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.380836    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:41.380982 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.380887    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:41.380982 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.380912    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:41.382482 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.382418    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb2e10b8-dd48-4a96-9715-63b3cc6874db-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:41.382482 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.382445    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eb2e10b8-dd48-4a96-9715-63b3cc6874db-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:41.382482 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.382458    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlq7v\" (UniqueName: \"kubernetes.io/projected/eb2e10b8-dd48-4a96-9715-63b3cc6874db-kube-api-access-qlq7v\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:41.383792 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.383772    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:41.387930 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.387898    4528 scope.go:122] "RemoveContainer" containerID="824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"
Oct 30 02:57:41.388257 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.388238    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96\": container with ID starting with 824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96 not found: ID does not exist" containerID="824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"
Oct 30 02:57:41.388355 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.388264    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96"} err="failed to get container status \"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96\": rpc error: code = NotFound desc = could not find container \"824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96\": container with ID starting with 824991fad8a3ef9e819b164103e5ac71da67cbcde0ca6e9b1319768cce8aea96 not found: ID does not exist"
Oct 30 02:57:41.391931 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.391913    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:41.421372 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.421347    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:41.427590 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.427564    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:41.471258 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.471193    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.643750366 podStartE2EDuration="16.47117633s" totalImagesPullingTime="6.827425964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:57:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:41.469982834 +0000 UTC m=+1019.694232114" watchObservedRunningTime="2026-10-30 02:57:41.47117633 +0000 UTC m=+1019.695425608"
Oct 30 02:57:41.483088 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483058    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:41.483268 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483137    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:41.983117099 +0000 UTC m=+1020.207366358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:41.483268 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483145    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:41.483268 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483232    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:41.983212955 +0000 UTC m=+1020.207462223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:41.483459 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483440    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:41.483530 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.483509    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:41.983491932 +0000 UTC m=+1020.207741200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:41.505121 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.505075    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:41.892030 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:41.891997    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" path="/var/lib/kubelet/pods/eb2e10b8-dd48-4a96-9715-63b3cc6874db/volumes"
Oct 30 02:57:41.987836 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987784    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:41.987836 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987806    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:41.987836 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987784    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:41.988044 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987865    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.987851214 +0000 UTC m=+1021.212100469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:41.988044 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987880    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.987873156 +0000 UTC m=+1021.212122412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:41.988044 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:41.987890    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.987884792 +0000 UTC m=+1021.212134048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:42.123983 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:42.123946    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:42.384903 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:42.384876    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-nfv94\" not found"
Oct 30 02:57:42.491511 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491472    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.491511 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491505    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491541    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.991523509 +0000 UTC m=+1021.215772790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491472    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491564    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.991552295 +0000 UTC m=+1021.215801553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491590    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491597    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.99158346 +0000 UTC m=+1021.215832727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:42.491962 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.491677    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:42.991654272 +0000 UTC m=+1021.215903531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:42.996410 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996360    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996438    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996370    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996449    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996387    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996387    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996449    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:43.996434505 +0000 UTC m=+1022.220683797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996473    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996552    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:43.996537928 +0000 UTC m=+1022.220787196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996568    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:44.996559024 +0000 UTC m=+1023.220808279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996589    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:43.996580808 +0000 UTC m=+1022.220830070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:42.996610 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996607    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:44.99660032 +0000 UTC m=+1023.220849575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:42.996976 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996625    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:43.996617414 +0000 UTC m=+1022.220866669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:42.996976 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:42.996642    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:44.996634781 +0000 UTC m=+1023.220884047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 02:57:43.388193 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:43.388134    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo" containerID="cri-o://60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397" gracePeriod=30
Oct 30 02:57:43.388193 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:43.388164    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway-opa" containerID="cri-o://0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604" gracePeriod=30
Oct 30 02:57:43.388720 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:43.388158    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway" containerID="cri-o://80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b" gracePeriod=30
Oct 30 02:57:44.006020 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.005971    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:44.006020 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.006003    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.005983    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.005993    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.006058    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:46.006042756 +0000 UTC m=+1024.230292012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.006104    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:46.006089096 +0000 UTC m=+1024.230338356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.006121    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:46.006114518 +0000 UTC m=+1024.230363773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 02:57:44.006224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:44.006134    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls podName:816655ab-6ab1-41b0-a303-de76659b0ad9 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:46.006129394 +0000 UTC m=+1024.230378650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 02:57:44.394450 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.394419    4528 generic.go:356] "Generic (PLEG): container finished" podID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerID="0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604" exitCode=0
Oct 30 02:57:44.394450 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.394444    4528 generic.go:356] "Generic (PLEG): container finished" podID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerID="60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397" exitCode=0
Oct 30 02:57:44.394867 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.394491    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerDied","Data":"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"}
Oct 30 02:57:44.394867 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.394545    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerDied","Data":"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"}
Oct 30 02:57:44.633851 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.633814    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:44.712055 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.711977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\" (UniqueName: \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712055 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712028    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712055 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712054    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712280 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712280 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712239    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712382 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712452 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712429    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712508 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712560 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\") pod \"816655ab-6ab1-41b0-a303-de76659b0ad9\" (UID: \"816655ab-6ab1-41b0-a303-de76659b0ad9\") "
Oct 30 02:57:44.712609 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712574    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:44.712714 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:44.712951 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712921    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.713069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712974    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.713069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:44.713069 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.712939    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:44.714445 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.714422    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:44.714580 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.714559    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "kube-api-access-csqqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:44.714748 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.714717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:44.714885 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.714867    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:44.716738 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.716718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "816655ab-6ab1-41b0-a303-de76659b0ad9" (UID: "816655ab-6ab1-41b0-a303-de76659b0ad9"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 02:57:44.813953 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813920    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 02:57:44.813953 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813951    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.813953 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813963    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.814183 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813973    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.814183 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813982    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqqs\" (UniqueName: \"kubernetes.io/projected/816655ab-6ab1-41b0-a303-de76659b0ad9-kube-api-access-csqqs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.814183 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.813991    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/816655ab-6ab1-41b0-a303-de76659b0ad9-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.814183 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.814000    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/816655ab-6ab1-41b0-a303-de76659b0ad9-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.824550 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.824527    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 02:57:44.914611 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:44.914579    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 02:57:45.400805 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.400774    4528 generic.go:356] "Generic (PLEG): container finished" podID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerID="80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b" exitCode=0
Oct 30 02:57:45.401386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.400847    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 02:57:45.401386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.400862    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerDied","Data":"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"}
Oct 30 02:57:45.401386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.400931    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"816655ab-6ab1-41b0-a303-de76659b0ad9","Type":"ContainerDied","Data":"39b690546a0929845e924f576b922e69e65a2fe7bdb33cce6933e103ced66e07"}
Oct 30 02:57:45.401386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.400965    4528 scope.go:122] "RemoveContainer" containerID="0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"
Oct 30 02:57:45.411961 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.409952    4528 scope.go:122] "RemoveContainer" containerID="80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"
Oct 30 02:57:45.421461 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.421435    4528 scope.go:122] "RemoveContainer" containerID="60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"
Oct 30 02:57:45.428334 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.428312    4528 scope.go:122] "RemoveContainer" containerID="0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"
Oct 30 02:57:45.428588 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:45.428569    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604\": container with ID starting with 0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604 not found: ID does not exist" containerID="0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"
Oct 30 02:57:45.428657 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.428600    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604"} err="failed to get container status \"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604\": rpc error: code = NotFound desc = could not find container \"0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604\": container with ID starting with 0f4ed02cbe5e93f66ee5e8ae1b9a8271473eeb1f724923b2351677c45057c604 not found: ID does not exist"
Oct 30 02:57:45.428657 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.428624    4528 scope.go:122] "RemoveContainer" containerID="80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"
Oct 30 02:57:45.428889 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:45.428867    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b\": container with ID starting with 80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b not found: ID does not exist" containerID="80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"
Oct 30 02:57:45.428983 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.428892    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b"} err="failed to get container status \"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b\": rpc error: code = NotFound desc = could not find container \"80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b\": container with ID starting with 80ec477c09f524df4d132c09f0fb85f5565ac8d8ae23134bc93d4dd200102c6b not found: ID does not exist"
Oct 30 02:57:45.428983 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.428906    4528 scope.go:122] "RemoveContainer" containerID="60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"
Oct 30 02:57:45.429120 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:45.429102    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397\": container with ID starting with 60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397 not found: ID does not exist" containerID="60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"
Oct 30 02:57:45.429159 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.429124    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397"} err="failed to get container status \"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397\": rpc error: code = NotFound desc = could not find container \"60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397\": container with ID starting with 60307b783dc86a252052a0d71fe67d84804fe2b37868f8b68f3106524f183397 not found: ID does not exist"
Oct 30 02:57:45.456039 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.456011    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:45.465958 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.465934    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 02:57:45.891682 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:45.891649    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" path="/var/lib/kubelet/pods/816655ab-6ab1-41b0-a303-de76659b0ad9/volumes"
Oct 30 02:57:46.116252 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:46.116219    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:46.250797 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:46.250713    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:46.383531 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:46.383490    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:47.249298 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249266    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249546    4528 cpu_manager.go:414] "containerMap: removing container" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway-opa"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249559    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway-opa"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249582    4528 cpu_manager.go:414] "containerMap: removing container" podUID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" containerName="otc-container"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249587    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" containerName="otc-container"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249605    4528 cpu_manager.go:414] "containerMap: removing container" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249613    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249623    4528 cpu_manager.go:414] "containerMap: removing container" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo"
Oct 30 02:57:47.249684 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249630    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo"
Oct 30 02:57:47.250098 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249699    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway"
Oct 30 02:57:47.250098 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249709    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo"
Oct 30 02:57:47.250098 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249715    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="816655ab-6ab1-41b0-a303-de76659b0ad9" containerName="tempo-gateway-opa"
Oct 30 02:57:47.250098 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.249721    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2e10b8-dd48-4a96-9715-63b3cc6874db" containerName="otc-container"
Oct 30 02:57:47.254418 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.254395    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.267357 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.267334    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-0df4277d\""
Oct 30 02:57:47.281936 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.281914    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:47.283313 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.283283    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:47.285527 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.285506    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 02:57:47.287305 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.287286    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 02:57:47.287386 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.287307    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-2sftv\""
Oct 30 02:57:47.333413 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.333383    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.333552 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.333418    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldwk\" (UniqueName: \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.333552 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.333523    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.333654 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.333572    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.342047 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.342022    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:57:47.342047 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.342034    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:57:47.432921 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.432889    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:57:47.434883 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.434858    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.434959 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.434891    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldwk\" (UniqueName: \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.434959 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.434918    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.434959 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.434940    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.435067 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:47.435035    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 02:57:47.435098 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:47.435092    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert podName:9d0c7aa8-6ce6-420e-8564-f408bbd56b48 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:47.935074489 +0000 UTC m=+1026.159323758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48") : secret "data-science-collector-ta-client-cert" not found
Oct 30 02:57:47.435488 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.435469    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.437314 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.437294    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.458941 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.458911    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldwk\" (UniqueName: \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.939831 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:47.939782    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:47.939997 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:47.939925    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found
Oct 30 02:57:47.939997 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:57:47.939987    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert podName:9d0c7aa8-6ce6-420e-8564-f408bbd56b48 nodeName:}" failed. No retries permitted until 2026-10-30 02:57:48.939970209 +0000 UTC m=+1027.164219478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48") : secret "data-science-collector-ta-client-cert" not found
Oct 30 02:57:48.949146 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:48.949110    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:48.951603 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:48.951573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:49.066473 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:49.066436    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 02:57:49.361995 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:49.361907    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 02:57:49.366253 ip-10-0-134-164 kubenswrapper[4528]: W1030 02:57:49.366225    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0c7aa8_6ce6_420e_8564_f408bbd56b48.slice/crio-45eafeb77e4afaeb64c0bd6d6bbfef080ab222f0c6af74a608c4bf92ab9ec1a4 WatchSource:0}: Error finding container 45eafeb77e4afaeb64c0bd6d6bbfef080ab222f0c6af74a608c4bf92ab9ec1a4: Status 404 returned error can't find the container with id 45eafeb77e4afaeb64c0bd6d6bbfef080ab222f0c6af74a608c4bf92ab9ec1a4
Oct 30 02:57:49.368788 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:49.368756    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:49.368900 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:49.368813    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 02:57:49.418888 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:49.418854    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9d0c7aa8-6ce6-420e-8564-f408bbd56b48","Type":"ContainerStarted","Data":"45eafeb77e4afaeb64c0bd6d6bbfef080ab222f0c6af74a608c4bf92ab9ec1a4"}
Oct 30 02:57:50.424253 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:50.424219    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9d0c7aa8-6ce6-420e-8564-f408bbd56b48","Type":"ContainerStarted","Data":"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"}
Oct 30 02:57:50.462188 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:50.462140    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.462128212 podStartE2EDuration="3.462128212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:50.460985557 +0000 UTC m=+1028.685234834" watchObservedRunningTime="2026-10-30 02:57:50.462128212 +0000 UTC m=+1028.686377489"
Oct 30 02:57:54.940775 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:57:54.940665    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:04.630885 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:58:04.630851    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:58:14.467778 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:58:14.467739    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:31.654396 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:58:31.654356    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:58:34.630704 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:58:34.630670    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:59:04.631224 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:59:04.631180    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:59:19.415485 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:59:19.415445    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:31.781721 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:59:31.781682    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 02:59:34.630793 ip-10-0-134-164 kubenswrapper[4528]: E1030 02:59:34.630760    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 02:59:37.911046 ip-10-0-134-164 kubenswrapper[4528]: I1030 02:59:37.911007    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:04.630593 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:00:04.630545    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:00:31.906700 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:00:31.906664    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:00:34.630479 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:00:34.630437    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:01:04.630464 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:01:04.630378    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:01:32.074491 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:01:32.074439    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:01:34.630318 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:01:34.630253    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:01:59.531374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:01:59.531338    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:04.630767 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:02:04.630721    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:02:09.234092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:02:09.234050    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:09.371055 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:02:09.371023    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:19.139105 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:02:19.139071    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:32.166697 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:02:32.166651    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:02:34.630412 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:02:34.630375    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:03:04.630723 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:03:04.630687    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:03:32.280535 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:03:32.280497    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:03:34.630654 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:03:34.630617    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:04:04.631254 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:04:04.631171    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:04:27.513645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:04:27.513604    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:32.451887 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:04:32.451851    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:04:34.629955 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:04:34.629919    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:05:04.630963 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:05:04.630930    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:05:32.546762 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:05:32.546706    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:34.629723 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:05:34.629687    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:05:58.784862 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:05:58.784805    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:05:58.922847 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:05:58.922799    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:04.630058 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:06:04.630022    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:06:32.669616 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:06:32.669581    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:06:34.632290 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:06:34.632246    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:07:04.631693 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:07:04.631611    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:07:32.790319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:07:32.790287    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:07:34.630929 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:07:34.630886    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:08:02.230964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.230929    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:02.231684 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.231138    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" containerName="otc-container" containerID="cri-o://37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55" gracePeriod=30
Oct 30 03:08:02.267471 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.267436    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:02.389954 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.389915    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:02.477965 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.477942    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:02.549970 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.549878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\") pod \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") "
Oct 30 03:08:02.549970 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.549962    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") pod \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") "
Oct 30 03:08:02.550208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.549990    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\" (UniqueName: \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\") pod \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") "
Oct 30 03:08:02.550208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.550039    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\") pod \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\" (UID: \"9d0c7aa8-6ce6-420e-8564-f408bbd56b48\") "
Oct 30 03:08:02.550555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.550510    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal" pod "9d0c7aa8-6ce6-420e-8564-f408bbd56b48" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:02.552224 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.552194    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk" pod "9d0c7aa8-6ce6-420e-8564-f408bbd56b48" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48"). InnerVolumeSpecName "kube-api-access-2ldwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:02.552494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.552470    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs" pod "9d0c7aa8-6ce6-420e-8564-f408bbd56b48" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:02.552494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.552482    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert" pod "9d0c7aa8-6ce6-420e-8564-f408bbd56b48" (UID: "9d0c7aa8-6ce6-420e-8564-f408bbd56b48"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:02.650990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.650955    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:02.650990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.650984    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:02.650990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.650994    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:02.651236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.651005    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldwk\" (UniqueName: \"kubernetes.io/projected/9d0c7aa8-6ce6-420e-8564-f408bbd56b48-kube-api-access-2ldwk\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:02.770218 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.770187    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" containerID="37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55" exitCode=0
Oct 30 03:08:02.770409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.770227    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9d0c7aa8-6ce6-420e-8564-f408bbd56b48","Type":"ContainerDied","Data":"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"}
Oct 30 03:08:02.770409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.770248    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:02.770409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.770269    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9d0c7aa8-6ce6-420e-8564-f408bbd56b48","Type":"ContainerDied","Data":"45eafeb77e4afaeb64c0bd6d6bbfef080ab222f0c6af74a608c4bf92ab9ec1a4"}
Oct 30 03:08:02.770409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.770290    4528 scope.go:122] "RemoveContainer" containerID="37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"
Oct 30 03:08:02.778523 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.778492    4528 scope.go:122] "RemoveContainer" containerID="37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"
Oct 30 03:08:02.778773 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:08:02.778750    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55\": container with ID starting with 37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55 not found: ID does not exist" containerID="37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"
Oct 30 03:08:02.778843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.778784    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55"} err="failed to get container status \"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55\": rpc error: code = NotFound desc = could not find container \"37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55\": container with ID starting with 37ad0a55aa495b94c76f0a58992390c3da9e2995da5e16f931d0f9ff236bdf55 not found: ID does not exist"
Oct 30 03:08:02.788666 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.788638    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:02.814584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.814516    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:02.818648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.818622    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:02.924724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:02.924688    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:03.892329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:03.892288    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" path="/var/lib/kubelet/pods/9d0c7aa8-6ce6-420e-8564-f408bbd56b48/volumes"
Oct 30 03:08:04.630311 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:08:04.630277    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:08:06.756789 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:06.756750    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:06.902228 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:06.902188    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:07.035579 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.035498    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:07.586055 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.586022    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:07.586348 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.586330    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" containerName="otc-container"
Oct 30 03:08:07.586348 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.586345    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" containerName="otc-container"
Oct 30 03:08:07.586486 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.586392    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0c7aa8-6ce6-420e-8564-f408bbd56b48" containerName="otc-container"
Oct 30 03:08:07.589150 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.589127    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.609935 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.609910    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:07.613830 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.613796    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:07.683247 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.683211    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:08:07.683247 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.683234    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-638a775f\""
Oct 30 03:08:07.683247 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.683235    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:08:07.683517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.683240    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:08:07.683517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.683301    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-x9jdm\""
Oct 30 03:08:07.688502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.688476    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.688628 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.688521    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.688628 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.688548    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.732917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.732889    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:07.789918 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.789888    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.790337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.789930    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.790337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.789951    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.790654 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.790631    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.792317 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.792295    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.812093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.812068    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\") pod \"data-science-collector-collector-1\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:07.901889 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:07.901862    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:08.198668 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.198597    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:08.204335 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:08:08.204307    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82814cb4_8ed2_4354_a6b7_5fc281b67d4f.slice/crio-f984cb12ea5cf78ce0c8640eb50cb40d3f5b34bbf2d2fe097edd495921caf485 WatchSource:0}: Error finding container f984cb12ea5cf78ce0c8640eb50cb40d3f5b34bbf2d2fe097edd495921caf485: Status 404 returned error can't find the container with id f984cb12ea5cf78ce0c8640eb50cb40d3f5b34bbf2d2fe097edd495921caf485
Oct 30 03:08:08.206334 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.206302    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:08.206436 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.206360    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:08.796963 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.796931    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"82814cb4-8ed2-4354-a6b7-5fc281b67d4f","Type":"ContainerStarted","Data":"2eceb67536e050564e351b78c0265e6ad0f3f97b1be3f5946eb705975e8e1be0"}
Oct 30 03:08:08.797327 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.796982    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"82814cb4-8ed2-4354-a6b7-5fc281b67d4f","Type":"ContainerStarted","Data":"f984cb12ea5cf78ce0c8640eb50cb40d3f5b34bbf2d2fe097edd495921caf485"}
Oct 30 03:08:08.833627 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:08.833563    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.833547162 podStartE2EDuration="1.833547162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:08.83350912 +0000 UTC m=+1647.057758402" watchObservedRunningTime="2026-10-30 03:08:08.833547162 +0000 UTC m=+1647.057796436"
Oct 30 03:08:22.628454 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.628415    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:22.628877 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.628673    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" containerName="otc-container" containerID="cri-o://2eceb67536e050564e351b78c0265e6ad0f3f97b1be3f5946eb705975e8e1be0" gracePeriod=30
Oct 30 03:08:22.704398 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.704365    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:22.835183 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.835142    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:22.854092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.854064    4528 generic.go:356] "Generic (PLEG): container finished" podID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" containerID="2eceb67536e050564e351b78c0265e6ad0f3f97b1be3f5946eb705975e8e1be0" exitCode=0
Oct 30 03:08:22.854203 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.854126    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"82814cb4-8ed2-4354-a6b7-5fc281b67d4f","Type":"ContainerDied","Data":"2eceb67536e050564e351b78c0265e6ad0f3f97b1be3f5946eb705975e8e1be0"}
Oct 30 03:08:22.869435 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.869412    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:22.910428 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.910359    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\") pod \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") "
Oct 30 03:08:22.910428 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.910417    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\") pod \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") "
Oct 30 03:08:22.910628 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.910522    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\") pod \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\" (UID: \"82814cb4-8ed2-4354-a6b7-5fc281b67d4f\") "
Oct 30 03:08:22.910779 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.910751    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal" pod "82814cb4-8ed2-4354-a6b7-5fc281b67d4f" (UID: "82814cb4-8ed2-4354-a6b7-5fc281b67d4f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:08:22.912564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.912537    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn" pod "82814cb4-8ed2-4354-a6b7-5fc281b67d4f" (UID: "82814cb4-8ed2-4354-a6b7-5fc281b67d4f"). InnerVolumeSpecName "kube-api-access-sh8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:08:22.912564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:22.912539    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs" pod "82814cb4-8ed2-4354-a6b7-5fc281b67d4f" (UID: "82814cb4-8ed2-4354-a6b7-5fc281b67d4f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:08:23.011462 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.011428    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8qn\" (UniqueName: \"kubernetes.io/projected/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-kube-api-access-sh8qn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:23.011462 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.011456    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:23.011462 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.011466    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/82814cb4-8ed2-4354-a6b7-5fc281b67d4f-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:08:23.150435 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.150396    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:23.260305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.260210    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:23.859459 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.859369    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"82814cb4-8ed2-4354-a6b7-5fc281b67d4f","Type":"ContainerDied","Data":"f984cb12ea5cf78ce0c8640eb50cb40d3f5b34bbf2d2fe097edd495921caf485"}
Oct 30 03:08:23.859459 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.859397    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:23.859459 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.859444    4528 scope.go:122] "RemoveContainer" containerID="2eceb67536e050564e351b78c0265e6ad0f3f97b1be3f5946eb705975e8e1be0"
Oct 30 03:08:23.903055 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.903023    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:23.913139 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:23.913116    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:25.891504 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:25.891464    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" path="/var/lib/kubelet/pods/82814cb4-8ed2-4354-a6b7-5fc281b67d4f/volumes"
Oct 30 03:08:26.012635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:26.012597    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:26.152554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:26.152454    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:26.269200 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:26.269164    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:27.141966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.141933    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:27.142427 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.142319    4528 cpu_manager.go:414] "containerMap: removing container" podUID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" containerName="otc-container"
Oct 30 03:08:27.142427 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.142337    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" containerName="otc-container"
Oct 30 03:08:27.142539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.142440    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="82814cb4-8ed2-4354-a6b7-5fc281b67d4f" containerName="otc-container"
Oct 30 03:08:27.147087 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.147064    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.171657 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.171628    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:27.172551 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.172509    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:08:27.172551 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.172544    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:08:27.175198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.175165    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:27.179292 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.179272    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-bbb0deae\""
Oct 30 03:08:27.182100 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.182082    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zlrvj\""
Oct 30 03:08:27.182203 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.182100    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:08:27.182203 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.182126    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:08:27.243569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.243535    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.243718 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.243584    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.243718 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.243681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzntm\" (UniqueName: \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.243799 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.243740    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.318151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.318111    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:27.344330 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.344301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.344481 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.344335    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.344481 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.344358    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzntm\" (UniqueName: \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.344481 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.344388    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.345073 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.345039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.347004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.346982    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.347083 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.347063    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.366276 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.366254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzntm\" (UniqueName: \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\") pod \"data-science-collector-collector-1\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.451421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.451316    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:27.459148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.459114    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:08:27.755230 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.755156    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:08:27.760109 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:08:27.760085    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be52f5a_9f75_4109_98e3_15a6402d0fd6.slice/crio-58e36d2213e47409d71f5aa3586e17777784e8c7fc8f41f11843c4c69b9bfcdb WatchSource:0}: Error finding container 58e36d2213e47409d71f5aa3586e17777784e8c7fc8f41f11843c4c69b9bfcdb: Status 404 returned error can't find the container with id 58e36d2213e47409d71f5aa3586e17777784e8c7fc8f41f11843c4c69b9bfcdb
Oct 30 03:08:27.762032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.761999    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:27.762117 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.762062    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:08:27.880596 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.880560    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5be52f5a-9f75-4109-98e3-15a6402d0fd6","Type":"ContainerStarted","Data":"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"}
Oct 30 03:08:27.880790 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.880616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5be52f5a-9f75-4109-98e3-15a6402d0fd6","Type":"ContainerStarted","Data":"58e36d2213e47409d71f5aa3586e17777784e8c7fc8f41f11843c4c69b9bfcdb"}
Oct 30 03:08:27.917708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:27.917664    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=0.917652319 podStartE2EDuration="917.652319ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:27.917472062 +0000 UTC m=+1666.141721339" watchObservedRunningTime="2026-10-30 03:08:27.917652319 +0000 UTC m=+1666.141901596"
Oct 30 03:08:32.906882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:08:32.906842    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:08:34.630718 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:08:34.630682    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:09:04.631392 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:09:04.631351    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:09:33.049734 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:09:33.049679    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:09:34.631061 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:09:34.631024    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:10:04.630212 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:10:04.630172    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:10:26.747927 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:10:26.747886    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:33.167655 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:10:33.167620    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:10:34.630173 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:10:34.630135    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:11:04.631238 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:11:04.631197    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:11:33.322664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:11:33.322628    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:11:34.630037 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:11:34.630003    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:11:34.989237 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:11:34.989149    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:04.631211 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:12:04.631168    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:12:25.737053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:12:25.737012    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:33.442753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:12:33.442712    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:12:34.631127 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:12:34.631089    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:12:36.502741 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:12:36.502701    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:04.631104 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:13:04.631025    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:13:09.528369 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:13:09.528332    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:33.554464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:13:33.554425    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:13:34.631640 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:13:34.631608    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:14:04.629927 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:14:04.629889    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:14:33.713478 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:14:33.713436    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:14:34.630432 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:14:34.630393    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:15:04.631535 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:15:04.631496    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:15:33.820151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:15:33.820113    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:15:34.630926 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:15:34.630891    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:16:04.630626 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:16:04.630552    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:16:33.927641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:16:33.927603    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:16:34.631212 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:16:34.631166    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:17:04.630490 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:17:04.630451    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:17:34.080279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:17:34.080248    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:17:34.631234 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:17:34.631202    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:17:51.314093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:17:51.314046    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:04.630193 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:18:04.630147    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:18:34.207221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:34.207169    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:34.630533 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:18:34.630499    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:18:41.882452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:41.882416    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:41.883017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:41.882600    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" containerName="otc-container" containerID="cri-o://032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c" gracePeriod=30
Oct 30 03:18:41.936874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:41.936814    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:42.054488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.054449    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:42.128964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.128938    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:42.154312 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.154246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\") pod \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") "
Oct 30 03:18:42.154312 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.154300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\") pod \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") "
Oct 30 03:18:42.154515 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.154317    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\") pod \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") "
Oct 30 03:18:42.154515 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.154372    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\" (UniqueName: \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\") pod \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\" (UID: \"5be52f5a-9f75-4109-98e3-15a6402d0fd6\") "
Oct 30 03:18:42.154688 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.154665    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal" pod "5be52f5a-9f75-4109-98e3-15a6402d0fd6" (UID: "5be52f5a-9f75-4109-98e3-15a6402d0fd6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:18:42.156411 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.156383    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert" pod "5be52f5a-9f75-4109-98e3-15a6402d0fd6" (UID: "5be52f5a-9f75-4109-98e3-15a6402d0fd6"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:42.156497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.156460    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm" pod "5be52f5a-9f75-4109-98e3-15a6402d0fd6" (UID: "5be52f5a-9f75-4109-98e3-15a6402d0fd6"). InnerVolumeSpecName "kube-api-access-gzntm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:42.156548 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.156534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs" pod "5be52f5a-9f75-4109-98e3-15a6402d0fd6" (UID: "5be52f5a-9f75-4109-98e3-15a6402d0fd6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:42.245874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.245811    4528 generic.go:356] "Generic (PLEG): container finished" podID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" containerID="032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c" exitCode=0
Oct 30 03:18:42.245874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.245849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5be52f5a-9f75-4109-98e3-15a6402d0fd6","Type":"ContainerDied","Data":"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"}
Oct 30 03:18:42.246090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.245937    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5be52f5a-9f75-4109-98e3-15a6402d0fd6","Type":"ContainerDied","Data":"58e36d2213e47409d71f5aa3586e17777784e8c7fc8f41f11843c4c69b9bfcdb"}
Oct 30 03:18:42.246090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.245949    4528 scope.go:122] "RemoveContainer" containerID="032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"
Oct 30 03:18:42.246090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.245937    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:42.254874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.254851    4528 scope.go:122] "RemoveContainer" containerID="032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"
Oct 30 03:18:42.255008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.254989    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.255071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.255016    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5be52f5a-9f75-4109-98e3-15a6402d0fd6-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.255071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.255032    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5be52f5a-9f75-4109-98e3-15a6402d0fd6-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.255071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.255046    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzntm\" (UniqueName: \"kubernetes.io/projected/5be52f5a-9f75-4109-98e3-15a6402d0fd6-kube-api-access-gzntm\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:18:42.255188 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:18:42.255113    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c\": container with ID starting with 032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c not found: ID does not exist" containerID="032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"
Oct 30 03:18:42.255188 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.255137    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c"} err="failed to get container status \"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c\": rpc error: code = NotFound desc = could not find container \"032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c\": container with ID starting with 032b45132b497268e5e3c998ffe3954cb9c493e99f78e30b89738b11bd7e417c not found: ID does not exist"
Oct 30 03:18:42.310870 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.310842    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:42.315625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.315600    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:42.509698 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.509595    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:42.629892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:42.629811    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:43.892095 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:43.892063    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" path="/var/lib/kubelet/pods/5be52f5a-9f75-4109-98e3-15a6402d0fd6/volumes"
Oct 30 03:18:48.018544 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.018502    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:48.164336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.164300    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:48.292024 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.291924    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:48.418829 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.418788    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:48.558291 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.558211    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:18:48.558510 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.558498    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" containerName="otc-container"
Oct 30 03:18:48.558547 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.558511    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" containerName="otc-container"
Oct 30 03:18:48.558583 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.558563    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be52f5a-9f75-4109-98e3-15a6402d0fd6" containerName="otc-container"
Oct 30 03:18:48.562783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.562767    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.564978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.564948    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:48.592006 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.591982    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:18:48.601690 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.601658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.601835 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.601694    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9ln2\" (UniqueName: \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.601893 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.601832    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.601955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.601938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.601992 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.601976    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.639692 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.639660    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:18:48.640180 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.640152    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-d5s62\""
Oct 30 03:18:48.661495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.661469    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:18:48.661622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.661515    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:18:48.661622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.661527    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:18:48.661622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.661604    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:18:48.661751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.661739    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:18:48.702854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.702799    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.702854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.702856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9ln2\" (UniqueName: \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.702900    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.702935    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.702955    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703478 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.703450    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.703483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.703584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.703573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.705364 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.705343    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.725601 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.725572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9ln2\" (UniqueName: \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-szvl6\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.877657 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.877568    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:18:48.926427 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:48.926389    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.044329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.044297    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.175448 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.175402    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:49.176529 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.176503    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:18:49.179429 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:18:49.179406    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51195c10_0663_4a57_8c7f_272bd8e8543a.slice/crio-9a0282e31dd3dee2407f34fb5021e6b493f116affd971c939a529f866b933b00 WatchSource:0}: Error finding container 9a0282e31dd3dee2407f34fb5021e6b493f116affd971c939a529f866b933b00: Status 404 returned error can't find the container with id 9a0282e31dd3dee2407f34fb5021e6b493f116affd971c939a529f866b933b00
Oct 30 03:18:49.181380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.181362    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:18:49.278235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.278200    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerStarted","Data":"9a0282e31dd3dee2407f34fb5021e6b493f116affd971c939a529f866b933b00"}
Oct 30 03:18:49.331351 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:49.331317    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.052971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.051522    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:50.055170 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.055147    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.079987 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.079937    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.086805 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.084993    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:18:50.086805 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.086718    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:50.114457 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.114421    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqfd\" (UniqueName: \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.114648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.114472    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.114648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.114504    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.114648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.114537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.114648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.114572    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.142715 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.142651    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:18:50.142715 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.142699    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:18:50.143016 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.142788    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-txljs\""
Oct 30 03:18:50.143225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.143186    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:18:50.207589 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.207551    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.215545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.215511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.215717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.215564    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.215717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.215600    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.215717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.215637    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.215919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.215744    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqfd\" (UniqueName: \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.216708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.216675    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.219055 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.219018    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.219198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.219172    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.219259 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.219213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.241045 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.240929    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqfd\" (UniqueName: \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\") pod \"data-science-collector-collector-1\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.340391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.340305    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.374536 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.374501    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:18:50.467836 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.467794    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:50.681344 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.681308    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:18:50.684866 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:18:50.684836    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a3d976_2449_43f9_b55e_08bb51902265.slice/crio-488b3dd06a6e964bb1aeabbbf7830af049dc4093ae660f274ecf0c10352381ec WatchSource:0}: Error finding container 488b3dd06a6e964bb1aeabbbf7830af049dc4093ae660f274ecf0c10352381ec: Status 404 returned error can't find the container with id 488b3dd06a6e964bb1aeabbbf7830af049dc4093ae660f274ecf0c10352381ec
Oct 30 03:18:50.687177 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.687149    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:50.687294 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:50.687220    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:18:51.293810 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:51.293776    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f7a3d976-2449-43f9-b55e-08bb51902265","Type":"ContainerStarted","Data":"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"}
Oct 30 03:18:51.294274 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:51.293853    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f7a3d976-2449-43f9-b55e-08bb51902265","Type":"ContainerStarted","Data":"488b3dd06a6e964bb1aeabbbf7830af049dc4093ae660f274ecf0c10352381ec"}
Oct 30 03:18:52.301522 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.301485    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerStarted","Data":"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"}
Oct 30 03:18:52.882796 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.882740    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.882724972 podStartE2EDuration="2.882724972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:51.338045506 +0000 UTC m=+2289.562294784" watchObservedRunningTime="2026-10-30 03:18:52.882724972 +0000 UTC m=+2291.106974249"
Oct 30 03:18:52.883511 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.883490    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:18:52.886876 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.886861    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:18:52.905570 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.905543    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 03:18:52.918566 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.918508    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:18:52.939883 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:52.939851    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jhn\" (UniqueName: \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl\" (UID: \"1a28c654-3e45-411d-a9f2-3e3563c2e906\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:18:53.041096 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.041054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jhn\" (UniqueName: \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl\" (UID: \"1a28c654-3e45-411d-a9f2-3e3563c2e906\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:18:53.064287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.064257    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jhn\" (UniqueName: \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl\" (UID: \"1a28c654-3e45-411d-a9f2-3e3563c2e906\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:18:53.103382 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.103329    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.198323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.198241    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:18:53.307492 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.307456    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerStarted","Data":"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"}
Oct 30 03:18:53.353175 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.353127    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" podStartSLOduration=1.935584915 podStartE2EDuration="5.353112693s" totalImagesPullingTime="3.417527778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:53.352969388 +0000 UTC m=+2291.577218665" watchObservedRunningTime="2026-10-30 03:18:53.353112693 +0000 UTC m=+2291.577362016"
Oct 30 03:18:53.370109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.370075    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.517590 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.517480    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:18:53.520127 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:18:53.520102    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a28c654_3e45_411d_a9f2_3e3563c2e906.slice/crio-252fd59c9cff914f0cd677b2a40510134c2a09f9bc34bddad76d32da07d789ac WatchSource:0}: Error finding container 252fd59c9cff914f0cd677b2a40510134c2a09f9bc34bddad76d32da07d789ac: Status 404 returned error can't find the container with id 252fd59c9cff914f0cd677b2a40510134c2a09f9bc34bddad76d32da07d789ac
Oct 30 03:18:53.527024 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.526995    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.750977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.750835    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.878610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.878583    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:53.997059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:53.997028    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:54.313681 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:54.313596    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" event={"ID":"1a28c654-3e45-411d-a9f2-3e3563c2e906","Type":"ContainerStarted","Data":"252fd59c9cff914f0cd677b2a40510134c2a09f9bc34bddad76d32da07d789ac"}
Oct 30 03:18:56.158915 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.158882    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:56.279671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.279634    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:56.324286 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.324250    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" event={"ID":"1a28c654-3e45-411d-a9f2-3e3563c2e906","Type":"ContainerStarted","Data":"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"}
Oct 30 03:18:56.363084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.363034    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" podStartSLOduration=2.285378737 podStartE2EDuration="4.363019556s" totalImagesPullingTime="2.077640819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:56.36192583 +0000 UTC m=+2294.586175108" watchObservedRunningTime="2026-10-30 03:18:56.363019556 +0000 UTC m=+2294.587268874"
Oct 30 03:18:56.854327 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.854292    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:18:56.857925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.857904    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.873284 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.873259    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:18:56.873284 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.873257    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:18:56.885103 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885080    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:18:56.885261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885150    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:18:56.885261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885198    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:18:56.885261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885199    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:18:56.885261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885220    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:18:56.885426 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.885282    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:18:56.886663 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.886642    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:18:56.959393 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.959360    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:56.973528 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.973492    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:18:56.973528 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.973525    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-mgknz\""
Oct 30 03:18:56.975917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.975884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfvd\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.975992 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.975940    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976022    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976119 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976084    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976119 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976111    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976182 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976133    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976207    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976229    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976280    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:56.976404 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:56.976334    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077192 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077154    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077192 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077195    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077230    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077288    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077328    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077355    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077404    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077773 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077773 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077482    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfvd\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.077773 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.077530    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.078170 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.078136    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.078420 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.078395    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.078584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.078475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.080550 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.080525    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081021 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.080972    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081021 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081144 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081059    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081144 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081169    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081256 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081239    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.081917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.081897    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:18:57.085552 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.085530    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:18:57.101834 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:18:57.101801    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfvd\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.609903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.609864    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.638011 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.637984    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.711069 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.711042    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.711204 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.711158    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.724802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.724774    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.902954 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.902917    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:00.926240 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:00.926214    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:01.080602 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:01.080564    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:01.418903 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:19:01.418866    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5304d55f_36d4_4e9d_8348_eb84085f3855.slice/crio-64a42ab230357e3928dc59a504a236fe625f26074fdbdd3de7f3b0dcc307810a WatchSource:0}: Error finding container 64a42ab230357e3928dc59a504a236fe625f26074fdbdd3de7f3b0dcc307810a: Status 404 returned error can't find the container with id 64a42ab230357e3928dc59a504a236fe625f26074fdbdd3de7f3b0dcc307810a
Oct 30 03:19:01.428939 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:01.428912    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:19:02.349242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:02.349207    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerStarted","Data":"64a42ab230357e3928dc59a504a236fe625f26074fdbdd3de7f3b0dcc307810a"}
Oct 30 03:19:04.359389 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:04.359349    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerStarted","Data":"52407e7f219623afc776d5012a7d2f7698a999a28de0d85fa5d078e6e46962b4"}
Oct 30 03:19:04.631289 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:19:04.631256    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:19:07.373071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:07.373037    4528 generic.go:356] "Generic (PLEG): container finished" podID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerID="52407e7f219623afc776d5012a7d2f7698a999a28de0d85fa5d078e6e46962b4" exitCode=0
Oct 30 03:19:07.373428 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:07.373107    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerDied","Data":"52407e7f219623afc776d5012a7d2f7698a999a28de0d85fa5d078e6e46962b4"}
Oct 30 03:19:11.394322 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:11.394237    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerStarted","Data":"ef015dced48c148c01867348435a33db090055d4850a9fc163328ed75bb6989d"}
Oct 30 03:19:11.394322 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:11.394303    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerStarted","Data":"70e122a64c682c61747eadf18ae25fedbb09e133898ffb9de0fc63158f249dc2"}
Oct 30 03:19:11.394734 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:11.394328    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerStarted","Data":"a2468fce6f2b8d3657999feef517496d4f923669ec1fdbc31721b21ebec03cdb"}
Oct 30 03:19:11.470670 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:11.470612    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.32198736 podStartE2EDuration="18.470597713s" totalImagesPullingTime="4.963890223s" totalInitContainerRuntime="3.18472013s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:18:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:11.457719857 +0000 UTC m=+2309.681969135" watchObservedRunningTime="2026-10-30 03:19:11.470597713 +0000 UTC m=+2309.694846989"
Oct 30 03:19:16.081779 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:16.081716    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:16.082267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:16.081837    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:16.087403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:16.087379    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:16.417941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:16.417908    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:19:34.328460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:19:34.328425    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:19:34.631930 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:19:34.631890    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:20:04.631010 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:20:04.630971    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:20:04.951699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:20:04.951613    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:07.125809 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:20:07.125774    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:34.458811 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:20:34.458772    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:20:34.631688 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:20:34.631650    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:21:04.631423 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:21:04.631385    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:21:34.602499 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:21:34.602457    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:21:34.630341 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:21:34.630305    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:22:04.630454 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:22:04.630376    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:22:34.631436 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:22:34.631396    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:22:34.671449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:22:34.671400    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:04.631886 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:23:04.631846    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:23:05.719898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:23:05.719858    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:23:34.631636 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:23:34.631595    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:23:34.836238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:23:34.836193    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:24:04.630542 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:24:04.630500    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:24:34.631451 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:24:34.631409    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:24:34.974932 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:24:34.974852    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:04.631615 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:25:04.631536    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:25:34.630939 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:25:34.630908    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:25:35.081761 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:25:35.081674    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:43.246596 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:25:43.246559    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.758256 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:25:58.758212    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:25:58.878790 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:25:58.878753    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:26:04.631016 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:26:04.630979    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:26:34.631195 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:26:34.631151    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:26:36.773827 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:26:36.773796    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:04.630481 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:27:04.630442    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:27:32.411036 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:27:32.411003    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:27:34.630783 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:27:34.630749    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:27:35.339780 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:27:35.339744    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:28:04.631545 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:28:04.631464    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:28:34.630427 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:28:34.630392    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:28:35.479258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:28:35.479225    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:04.630893 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:04.630862    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:29:20.915882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:20.915840    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:29:20.916508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:20.916082    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="kube-rbac-proxy" containerID="cri-o://3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2" gracePeriod=30
Oct 30 03:29:20.916508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:20.916143    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="prom-label-proxy" containerID="cri-o://23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975" gracePeriod=30
Oct 30 03:29:21.153975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.153948    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:29:21.252551 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.252464    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\") pod \"51195c10-0663-4a57-8c7f-272bd8e8543a\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") "
Oct 30 03:29:21.252702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.252553    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\") pod \"51195c10-0663-4a57-8c7f-272bd8e8543a\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") "
Oct 30 03:29:21.252702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.252612    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\" (UniqueName: \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\") pod \"51195c10-0663-4a57-8c7f-272bd8e8543a\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") "
Oct 30 03:29:21.252702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.252656    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\") pod \"51195c10-0663-4a57-8c7f-272bd8e8543a\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") "
Oct 30 03:29:21.252888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.252715    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\" (UniqueName: \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\") pod \"51195c10-0663-4a57-8c7f-272bd8e8543a\" (UID: \"51195c10-0663-4a57-8c7f-272bd8e8543a\") "
Oct 30 03:29:21.253123 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.253085    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca" pod "51195c10-0663-4a57-8c7f-272bd8e8543a" (UID: "51195c10-0663-4a57-8c7f-272bd8e8543a"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:21.253386 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.253360    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config" pod "51195c10-0663-4a57-8c7f-272bd8e8543a" (UID: "51195c10-0663-4a57-8c7f-272bd8e8543a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:21.253651 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.253627    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca" pod "51195c10-0663-4a57-8c7f-272bd8e8543a" (UID: "51195c10-0663-4a57-8c7f-272bd8e8543a"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:21.255545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.255522    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2" pod "51195c10-0663-4a57-8c7f-272bd8e8543a" (UID: "51195c10-0663-4a57-8c7f-272bd8e8543a"). InnerVolumeSpecName "kube-api-access-r9ln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:21.255640 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.255604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls" pod "51195c10-0663-4a57-8c7f-272bd8e8543a" (UID: "51195c10-0663-4a57-8c7f-272bd8e8543a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:21.260593 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.260565    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:29:21.260860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.260803    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" podUID="1a28c654-3e45-411d-a9f2-3e3563c2e906" containerName="thanos-querier" containerID="cri-o://fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa" gracePeriod=30
Oct 30 03:29:21.334753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.334716    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.353848 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.353804    4528 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.353848 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.353848    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/51195c10-0663-4a57-8c7f-272bd8e8543a-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.354030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.353862    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.354030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.353876    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9ln2\" (UniqueName: \"kubernetes.io/projected/51195c10-0663-4a57-8c7f-272bd8e8543a-kube-api-access-r9ln2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.354030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.353891    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/51195c10-0663-4a57-8c7f-272bd8e8543a-prometheus-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.422164 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.421907    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:29:21.422360 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.422227    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="prometheus" containerID="cri-o://a2468fce6f2b8d3657999feef517496d4f923669ec1fdbc31721b21ebec03cdb" gracePeriod=600
Oct 30 03:29:21.422597 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.422571    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="thanos-sidecar" containerID="cri-o://ef015dced48c148c01867348435a33db090055d4850a9fc163328ed75bb6989d" gracePeriod=600
Oct 30 03:29:21.422725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.422693    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="config-reloader" containerID="cri-o://70e122a64c682c61747eadf18ae25fedbb09e133898ffb9de0fc63158f249dc2" gracePeriod=600
Oct 30 03:29:21.465898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.465859    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.537442 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.537415    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:29:21.584280 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.584244    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.658295 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.658265    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\" (UniqueName: \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\") pod \"1a28c654-3e45-411d-a9f2-3e3563c2e906\" (UID: \"1a28c654-3e45-411d-a9f2-3e3563c2e906\") "
Oct 30 03:29:21.660192 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.660167    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn" pod "1a28c654-3e45-411d-a9f2-3e3563c2e906" (UID: "1a28c654-3e45-411d-a9f2-3e3563c2e906"). InnerVolumeSpecName "kube-api-access-25jhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:21.704889 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.704852    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:21.758955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.758906    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jhn\" (UniqueName: \"kubernetes.io/projected/1a28c654-3e45-411d-a9f2-3e3563c2e906-kube-api-access-25jhn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:21.888513 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.888478    4528 generic.go:356] "Generic (PLEG): container finished" podID="1a28c654-3e45-411d-a9f2-3e3563c2e906" containerID="fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa" exitCode=0
Oct 30 03:29:21.889185 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.889162    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"
Oct 30 03:29:21.890225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.890201    4528 generic.go:356] "Generic (PLEG): container finished" podID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerID="23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975" exitCode=0
Oct 30 03:29:21.890225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.890223    4528 generic.go:356] "Generic (PLEG): container finished" podID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerID="3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2" exitCode=0
Oct 30 03:29:21.890398 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.890332    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"
Oct 30 03:29:21.893372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893343    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" event={"ID":"1a28c654-3e45-411d-a9f2-3e3563c2e906","Type":"ContainerDied","Data":"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"}
Oct 30 03:29:21.893508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893385    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl" event={"ID":"1a28c654-3e45-411d-a9f2-3e3563c2e906","Type":"ContainerDied","Data":"252fd59c9cff914f0cd677b2a40510134c2a09f9bc34bddad76d32da07d789ac"}
Oct 30 03:29:21.893508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerDied","Data":"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"}
Oct 30 03:29:21.893508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893438    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerDied","Data":"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"}
Oct 30 03:29:21.893508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893465    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6" event={"ID":"51195c10-0663-4a57-8c7f-272bd8e8543a","Type":"ContainerDied","Data":"9a0282e31dd3dee2407f34fb5021e6b493f116affd971c939a529f866b933b00"}
Oct 30 03:29:21.893508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893479    4528 scope.go:122] "RemoveContainer" containerID="fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"
Oct 30 03:29:21.893764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893655    4528 generic.go:356] "Generic (PLEG): container finished" podID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerID="ef015dced48c148c01867348435a33db090055d4850a9fc163328ed75bb6989d" exitCode=0
Oct 30 03:29:21.893764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893674    4528 generic.go:356] "Generic (PLEG): container finished" podID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerID="70e122a64c682c61747eadf18ae25fedbb09e133898ffb9de0fc63158f249dc2" exitCode=0
Oct 30 03:29:21.893764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893680    4528 generic.go:356] "Generic (PLEG): container finished" podID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerID="a2468fce6f2b8d3657999feef517496d4f923669ec1fdbc31721b21ebec03cdb" exitCode=0
Oct 30 03:29:21.893764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893730    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerDied","Data":"ef015dced48c148c01867348435a33db090055d4850a9fc163328ed75bb6989d"}
Oct 30 03:29:21.893998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893795    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerDied","Data":"70e122a64c682c61747eadf18ae25fedbb09e133898ffb9de0fc63158f249dc2"}
Oct 30 03:29:21.893998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.893848    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerDied","Data":"a2468fce6f2b8d3657999feef517496d4f923669ec1fdbc31721b21ebec03cdb"}
Oct 30 03:29:21.903251 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.903161    4528 scope.go:122] "RemoveContainer" containerID="fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"
Oct 30 03:29:21.903489 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:21.903458    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa\": container with ID starting with fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa not found: ID does not exist" containerID="fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"
Oct 30 03:29:21.903586 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.903498    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa"} err="failed to get container status \"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa\": rpc error: code = NotFound desc = could not find container \"fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa\": container with ID starting with fc8ac95cd461f45b6c5d9800defa4ae9c56e5d9b6f2119b7bb89d8cc6a6a45fa not found: ID does not exist"
Oct 30 03:29:21.903586 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.903517    4528 scope.go:122] "RemoveContainer" containerID="23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"
Oct 30 03:29:21.911145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.911119    4528 scope.go:122] "RemoveContainer" containerID="3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"
Oct 30 03:29:21.918636 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.918448    4528 scope.go:122] "RemoveContainer" containerID="23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"
Oct 30 03:29:21.918918 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:21.918710    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975\": container with ID starting with 23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975 not found: ID does not exist" containerID="23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"
Oct 30 03:29:21.918918 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.918730    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"} err="failed to get container status \"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975\": rpc error: code = NotFound desc = could not find container \"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975\": container with ID starting with 23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975 not found: ID does not exist"
Oct 30 03:29:21.918918 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.918746    4528 scope.go:122] "RemoveContainer" containerID="3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"
Oct 30 03:29:21.919061 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:21.919036    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2\": container with ID starting with 3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2 not found: ID does not exist" containerID="3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"
Oct 30 03:29:21.919097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.919072    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"} err="failed to get container status \"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2\": rpc error: code = NotFound desc = could not find container \"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2\": container with ID starting with 3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2 not found: ID does not exist"
Oct 30 03:29:21.919133 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.919094    4528 scope.go:122] "RemoveContainer" containerID="23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"
Oct 30 03:29:21.919341 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.919324    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975"} err="failed to get container status \"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975\": rpc error: code = NotFound desc = could not find container \"23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975\": container with ID starting with 23d0161e6e0318bfb677c37ccbf8dcd88df70a5b4316c16734b3fdba7b124975 not found: ID does not exist"
Oct 30 03:29:21.919389 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.919341    4528 scope.go:122] "RemoveContainer" containerID="3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"
Oct 30 03:29:21.919566 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.919547    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2"} err="failed to get container status \"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2\": rpc error: code = NotFound desc = could not find container \"3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2\": container with ID starting with 3ad541288d905fea10f0ff11a0471e6af58735c41fcbcc40f98df3a5c7a061a2 not found: ID does not exist"
Oct 30 03:29:21.946676 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.946643    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:29:21.952859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.952836    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-cg7wl"]
Oct 30 03:29:21.982809 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.982781    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:29:21.987480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:21.987454    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-szvl6"]
Oct 30 03:29:22.096478 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.096446    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:22.096735 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.096707    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f7a3d976-2449-43f9-b55e-08bb51902265" containerName="otc-container" containerID="cri-o://3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b" gracePeriod=30
Oct 30 03:29:22.149477 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.149400    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:22.228713 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.228689    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:29:22.276317 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.276289    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:22.342617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.342593    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:22.365015 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.364984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365109    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365159    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365185    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365224    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365259    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365374    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365432    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365525    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365570    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365634    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365667    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"5304d55f-36d4-4e9d-8348-eb84085f3855\" (UID: \"5304d55f-36d4-4e9d-8348-eb84085f3855\") "
Oct 30 03:29:22.365846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365766    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:22.366170 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.365957    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.366425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.366401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:22.366615 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.366589    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:22.367728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.367693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.368333 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.368305    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.368417 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.368324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.368671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.368646    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "kube-api-access-slfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:22.368898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.368870    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.369471 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.369446    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.369698 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.369679    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:22.369780 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.369680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:22.370754 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.370736    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:29:22.379391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.379369    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config" pod "5304d55f-36d4-4e9d-8348-eb84085f3855" (UID: "5304d55f-36d4-4e9d-8348-eb84085f3855"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.396159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.396117    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:22.466506 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466418    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\") pod \"f7a3d976-2449-43f9-b55e-08bb51902265\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") "
Oct 30 03:29:22.466506 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466473    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\") pod \"f7a3d976-2449-43f9-b55e-08bb51902265\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") "
Oct 30 03:29:22.466733 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466533    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\" (UniqueName: \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\") pod \"f7a3d976-2449-43f9-b55e-08bb51902265\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") "
Oct 30 03:29:22.466733 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466572    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\") pod \"f7a3d976-2449-43f9-b55e-08bb51902265\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") "
Oct 30 03:29:22.466733 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466611    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\") pod \"f7a3d976-2449-43f9-b55e-08bb51902265\" (UID: \"f7a3d976-2449-43f9-b55e-08bb51902265\") "
Oct 30 03:29:22.466929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466848    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5304d55f-36d4-4e9d-8348-eb84085f3855-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.466929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466868    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.466929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466884    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.466929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466917    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466933    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfvd\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-kube-api-access-slfvd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466945    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466954    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5304d55f-36d4-4e9d-8348-eb84085f3855-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466965    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466979    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.466994    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5304d55f-36d4-4e9d-8348-eb84085f3855-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.467007    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.467007    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal" pod "f7a3d976-2449-43f9-b55e-08bb51902265" (UID: "f7a3d976-2449-43f9-b55e-08bb51902265"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:29:22.467138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.467020    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5304d55f-36d4-4e9d-8348-eb84085f3855-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.468723 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.468700    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd" pod "f7a3d976-2449-43f9-b55e-08bb51902265" (UID: "f7a3d976-2449-43f9-b55e-08bb51902265"). InnerVolumeSpecName "kube-api-access-shqfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:22.469033 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.469015    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert" pod "f7a3d976-2449-43f9-b55e-08bb51902265" (UID: "f7a3d976-2449-43f9-b55e-08bb51902265"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.469091 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.469040    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs" pod "f7a3d976-2449-43f9-b55e-08bb51902265" (UID: "f7a3d976-2449-43f9-b55e-08bb51902265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.469230 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.469209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs" pod "f7a3d976-2449-43f9-b55e-08bb51902265" (UID: "f7a3d976-2449-43f9-b55e-08bb51902265"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:22.478007 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.477992    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:29:22.568275 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568239    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.568275 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568269    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqfd\" (UniqueName: \"kubernetes.io/projected/f7a3d976-2449-43f9-b55e-08bb51902265-kube-api-access-shqfd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.568275 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568281    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f7a3d976-2449-43f9-b55e-08bb51902265-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.568568 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568289    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.568568 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568299    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 03:29:22.568568 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.568308    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7a3d976-2449-43f9-b55e-08bb51902265-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:29:22.898837 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.898789    4528 generic.go:356] "Generic (PLEG): container finished" podID="f7a3d976-2449-43f9-b55e-08bb51902265" containerID="3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b" exitCode=0
Oct 30 03:29:22.899018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.898861    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:29:22.899018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.898873    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f7a3d976-2449-43f9-b55e-08bb51902265","Type":"ContainerDied","Data":"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"}
Oct 30 03:29:22.899018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.898929    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f7a3d976-2449-43f9-b55e-08bb51902265","Type":"ContainerDied","Data":"488b3dd06a6e964bb1aeabbbf7830af049dc4093ae660f274ecf0c10352381ec"}
Oct 30 03:29:22.899018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.898957    4528 scope.go:122] "RemoveContainer" containerID="3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"
Oct 30 03:29:22.903642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.903626    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:29:22.903745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.903622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"5304d55f-36d4-4e9d-8348-eb84085f3855","Type":"ContainerDied","Data":"64a42ab230357e3928dc59a504a236fe625f26074fdbdd3de7f3b0dcc307810a"}
Oct 30 03:29:22.907687 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.907668    4528 scope.go:122] "RemoveContainer" containerID="3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"
Oct 30 03:29:22.908048 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:22.907970    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b\": container with ID starting with 3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b not found: ID does not exist" containerID="3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"
Oct 30 03:29:22.908048 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.907998    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b"} err="failed to get container status \"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b\": rpc error: code = NotFound desc = could not find container \"3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b\": container with ID starting with 3c72b22bd368cc260574605e0d8fdf7ae6ff9b62802e25cf904c734f0070698b not found: ID does not exist"
Oct 30 03:29:22.908048 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.908018    4528 scope.go:122] "RemoveContainer" containerID="ef015dced48c148c01867348435a33db090055d4850a9fc163328ed75bb6989d"
Oct 30 03:29:22.916394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.916375    4528 scope.go:122] "RemoveContainer" containerID="70e122a64c682c61747eadf18ae25fedbb09e133898ffb9de0fc63158f249dc2"
Oct 30 03:29:22.924242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.924104    4528 scope.go:122] "RemoveContainer" containerID="a2468fce6f2b8d3657999feef517496d4f923669ec1fdbc31721b21ebec03cdb"
Oct 30 03:29:22.943120 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.943099    4528 scope.go:122] "RemoveContainer" containerID="52407e7f219623afc776d5012a7d2f7698a999a28de0d85fa5d078e6e46962b4"
Oct 30 03:29:22.943525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.943505    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:22.953562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.953538    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:29:22.998108 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:22.998078    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:29:23.006326 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.006298    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:29:23.145684 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.145648    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.261458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.261366    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.378264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.378223    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:23.892081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.891989    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a28c654-3e45-411d-a9f2-3e3563c2e906" path="/var/lib/kubelet/pods/1a28c654-3e45-411d-a9f2-3e3563c2e906/volumes"
Oct 30 03:29:23.892456 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.892436    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" path="/var/lib/kubelet/pods/51195c10-0663-4a57-8c7f-272bd8e8543a/volumes"
Oct 30 03:29:23.892985 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.892965    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" path="/var/lib/kubelet/pods/5304d55f-36d4-4e9d-8348-eb84085f3855/volumes"
Oct 30 03:29:23.893774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:23.893752    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a3d976-2449-43f9-b55e-08bb51902265" path="/var/lib/kubelet/pods/f7a3d976-2449-43f9-b55e-08bb51902265/volumes"
Oct 30 03:29:26.431158 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:26.431124    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:26.562458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:26.562413    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:26.680030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:26.679995    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:26.797969 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:26.797878    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:27.275324 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275291    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:27.275637 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275614    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f7a3d976-2449-43f9-b55e-08bb51902265" containerName="otc-container"
Oct 30 03:29:27.275637 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275632    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a3d976-2449-43f9-b55e-08bb51902265" containerName="otc-container"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275651    4528 cpu_manager.go:414] "containerMap: removing container" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="kube-rbac-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275659    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="kube-rbac-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275668    4528 cpu_manager.go:414] "containerMap: removing container" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="prom-label-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275674    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="prom-label-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275691    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1a28c654-3e45-411d-a9f2-3e3563c2e906" containerName="thanos-querier"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275698    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a28c654-3e45-411d-a9f2-3e3563c2e906" containerName="thanos-querier"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275713    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="prometheus"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275719    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="prometheus"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275729    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="config-reloader"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275736    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="config-reloader"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275747    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="thanos-sidecar"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275753    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="thanos-sidecar"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275765    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="init-config-reloader"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275772    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="init-config-reloader"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275876    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a3d976-2449-43f9-b55e-08bb51902265" containerName="otc-container"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275888    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="kube-rbac-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275896    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="51195c10-0663-4a57-8c7f-272bd8e8543a" containerName="prom-label-proxy"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275904    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a28c654-3e45-411d-a9f2-3e3563c2e906" containerName="thanos-querier"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275914    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="prometheus"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275923    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="config-reloader"
Oct 30 03:29:27.276081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.275932    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304d55f-36d4-4e9d-8348-eb84085f3855" containerName="thanos-sidecar"
Oct 30 03:29:27.280688 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.280666    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.295641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.295620    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:29:27.295793 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.295727    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kxk4q\""
Oct 30 03:29:27.304794 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.304762    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.304958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.304808    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.304958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.304873    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddsmb\" (UniqueName: \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.305924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.305893    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:29:27.307512 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.307488    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:27.333381 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.333347    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:27.397226 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.397196    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:29:27.397571 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.397554    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 03:29:27.405897 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.405868    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.406000 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.405910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.406000 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.405949    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddsmb\" (UniqueName: \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.406635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.406615    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.408374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.408357    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.428367 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.428341    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddsmb\" (UniqueName: \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\") pod \"data-science-collector-collector-0\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.457083 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.457051    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:27.586441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.586341    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:27.595173 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.595138    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:29:27.891623 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.891592    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:29:27.896380 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:29:27.896350    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4935057_09e7_4c80_b11e_c193928140c5.slice/crio-630df5d960e730a38adbe844e8ac01fb723b9976855791db775f4e8fba20f2c7 WatchSource:0}: Error finding container 630df5d960e730a38adbe844e8ac01fb723b9976855791db775f4e8fba20f2c7: Status 404 returned error can't find the container with id 630df5d960e730a38adbe844e8ac01fb723b9976855791db775f4e8fba20f2c7
Oct 30 03:29:27.898394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.898364    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:29:27.898508 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.898432    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:29:27.927352 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:27.927323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d4935057-09e7-4c80-b11e-c193928140c5","Type":"ContainerStarted","Data":"630df5d960e730a38adbe844e8ac01fb723b9976855791db775f4e8fba20f2c7"}
Oct 30 03:29:28.466899 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.466868    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:29:28.470609 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.470589    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.485499 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.485474    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:29:28.485960 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.485937    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:29:28.485960 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.485937    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:29:28.486149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.485982    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:29:28.486149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.485986    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:29:28.486149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.486053    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:29:28.486149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.486085    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:29:28.495637 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.495613    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-cd6ml\""
Oct 30 03:29:28.497563 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.497067    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:29:28.520103 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.520069    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:28.616440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616442    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616691 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616531    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616691 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616569    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cwn\" (UniqueName: \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616691 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616629    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616691 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616650    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616864 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616714    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.616864 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.616739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.643286 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.643245    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:28.717781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717693    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.717781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717731    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.717781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717760    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.717781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717783    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cwn\" (UniqueName: \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717805    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717848    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717892    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.717929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718594 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.718564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.718684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.718699    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.718945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.718782    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.720414 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.720384    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.720520 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.720431    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.720520 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.720453    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.738998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.738977    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cwn\" (UniqueName: \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:28.933839 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.933785    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d4935057-09e7-4c80-b11e-c193928140c5","Type":"ContainerStarted","Data":"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"}
Oct 30 03:29:28.971673 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:28.971577    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.971563544 podStartE2EDuration="1.971563544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:28.970601219 +0000 UTC m=+2927.194850498" watchObservedRunningTime="2026-10-30 03:29:28.971563544 +0000 UTC m=+2927.195812815"
Oct 30 03:29:33.356115 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.356080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.380615 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.380589    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.456539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.456506    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.456676 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.456637    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.468892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.468870    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.568096 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.568066    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.592713 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.592683    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:33.886222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:33.886182    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:34.224170 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.224037    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:29:34.631592 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:29:34.631560    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:29:34.963849 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.963728    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerStarted","Data":"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"}
Oct 30 03:29:34.963849 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.963793    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerStarted","Data":"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"}
Oct 30 03:29:34.963849 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.963848    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:34.964088 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.963872    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerStarted","Data":"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"}
Oct 30 03:29:34.964088 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.963894    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerStarted","Data":"db04d8db7704e21272433d0f44cf4e6e7e2783bed80f062dcf76748f523f83ca"}
Oct 30 03:29:34.966369 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:34.966347    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:35.025407 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.025347    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.025334548 podStartE2EDuration="7.025334548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:35.015144608 +0000 UTC m=+2933.239393886" watchObservedRunningTime="2026-10-30 03:29:35.025334548 +0000 UTC m=+2933.249583872"
Oct 30 03:29:35.586592 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.586558    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:35.750619 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.750585    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:29:35.967812 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.967778    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:35.968078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.968062    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:35.976540 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:35.976518    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:29:57.978526 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:29:57.978481    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:30:04.630690 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:04.630651    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:30:07.978217 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:07.978172    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:30:17.977646 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:17.977597    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:20.568065 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.568027    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:30:20.568521 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.568293    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d4935057-09e7-4c80-b11e-c193928140c5" containerName="otc-container" containerID="cri-o://fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d" gracePeriod=30
Oct 30 03:30:20.653334 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.653288    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:20.773161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.773118    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:20.814587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.814553    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:30:20.890919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.890883    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:20.934275 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.934242    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\" (UniqueName: \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\") pod \"d4935057-09e7-4c80-b11e-c193928140c5\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") "
Oct 30 03:30:20.934452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.934287    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\") pod \"d4935057-09e7-4c80-b11e-c193928140c5\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") "
Oct 30 03:30:20.934452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.934323    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\") pod \"d4935057-09e7-4c80-b11e-c193928140c5\" (UID: \"d4935057-09e7-4c80-b11e-c193928140c5\") "
Oct 30 03:30:20.934755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.934718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal" pod "d4935057-09e7-4c80-b11e-c193928140c5" (UID: "d4935057-09e7-4c80-b11e-c193928140c5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.936380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.936361    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb" pod "d4935057-09e7-4c80-b11e-c193928140c5" (UID: "d4935057-09e7-4c80-b11e-c193928140c5"). InnerVolumeSpecName "kube-api-access-ddsmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.936458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:20.936443    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs" pod "d4935057-09e7-4c80-b11e-c193928140c5" (UID: "d4935057-09e7-4c80-b11e-c193928140c5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:21.035221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.035190    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d4935057-09e7-4c80-b11e-c193928140c5-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.035221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.035219    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddsmb\" (UniqueName: \"kubernetes.io/projected/d4935057-09e7-4c80-b11e-c193928140c5-kube-api-access-ddsmb\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.035221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.035230    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4935057-09e7-4c80-b11e-c193928140c5-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:21.060903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.060871    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:21.061144 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.061100    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo" containerID="cri-o://2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1" gracePeriod=30
Oct 30 03:30:21.061460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.061435    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway-opa" containerID="cri-o://67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448" gracePeriod=30
Oct 30 03:30:21.061567 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.061546    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway" containerID="cri-o://2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0" gracePeriod=30
Oct 30 03:30:21.130373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.130324    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:21.156423 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.156329    4528 generic.go:356] "Generic (PLEG): container finished" podID="d4935057-09e7-4c80-b11e-c193928140c5" containerID="fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d" exitCode=0
Oct 30 03:30:21.156423 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.156409    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d4935057-09e7-4c80-b11e-c193928140c5","Type":"ContainerDied","Data":"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"}
Oct 30 03:30:21.156602 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.156488    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d4935057-09e7-4c80-b11e-c193928140c5","Type":"ContainerDied","Data":"630df5d960e730a38adbe844e8ac01fb723b9976855791db775f4e8fba20f2c7"}
Oct 30 03:30:21.156602 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.156515    4528 scope.go:122] "RemoveContainer" containerID="fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"
Oct 30 03:30:21.156692 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.156666    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 03:30:21.173059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.173036    4528 scope.go:122] "RemoveContainer" containerID="fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"
Oct 30 03:30:21.173403 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:21.173387    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d\": container with ID starting with fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d not found: ID does not exist" containerID="fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"
Oct 30 03:30:21.173506 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.173412    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d"} err="failed to get container status \"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d\": rpc error: code = NotFound desc = could not find container \"fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d\": container with ID starting with fa6bf87b65f9249a7e3b10c2cabf0591e7c622b29aa7615e957a01b9af4a3f4d not found: ID does not exist"
Oct 30 03:30:21.202022 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.201996    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:30:21.206046 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.206020    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 03:30:21.270746 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.270716    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:21.893184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:21.893152    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4935057-09e7-4c80-b11e-c193928140c5" path="/var/lib/kubelet/pods/d4935057-09e7-4c80-b11e-c193928140c5/volumes"
Oct 30 03:30:22.162627 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.162599    4528 generic.go:356] "Generic (PLEG): container finished" podID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerID="67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448" exitCode=0
Oct 30 03:30:22.162627 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.162621    4528 generic.go:356] "Generic (PLEG): container finished" podID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerID="2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1" exitCode=0
Oct 30 03:30:22.162783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.162671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerDied","Data":"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"}
Oct 30 03:30:22.162783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.162718    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerDied","Data":"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"}
Oct 30 03:30:22.312482 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.312453    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:22.447907 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.447801    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.447907 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.447882    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448017    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448137 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448079    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448175 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448137    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448177    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448207    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448384 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448362    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448448 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448436    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\" (UniqueName: \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\") pod \"96575aae-7083-46a5-9a3e-fd7864e8f987\" (UID: \"96575aae-7083-46a5-9a3e-fd7864e8f987\") "
Oct 30 03:30:22.448546 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448423    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.448623 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448595    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.448728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448231    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.448793 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448771    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.448866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448793    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.448866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448807    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.449009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.448898    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:22.450350 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.450317    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:22.450528 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.450507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:22.450983 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.450961    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:22.450983 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.450966    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "kube-api-access-j7cwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:22.451208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.451187    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "96575aae-7083-46a5-9a3e-fd7864e8f987" (UID: "96575aae-7083-46a5-9a3e-fd7864e8f987"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:30:22.549207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549172    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.549207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549202    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.549207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549211    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.549438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549221    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7cwn\" (UniqueName: \"kubernetes.io/projected/96575aae-7083-46a5-9a3e-fd7864e8f987-kube-api-access-j7cwn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.549438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549232    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/96575aae-7083-46a5-9a3e-fd7864e8f987-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:30:22.549438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.549260    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:30:22.557358 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.557337    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:30:22.650289 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:22.650255    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:30:23.170205 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.170169    4528 generic.go:356] "Generic (PLEG): container finished" podID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerID="2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0" exitCode=0
Oct 30 03:30:23.170649 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.170225    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:23.170649 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.170230    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerDied","Data":"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"}
Oct 30 03:30:23.170649 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.170274    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"96575aae-7083-46a5-9a3e-fd7864e8f987","Type":"ContainerDied","Data":"db04d8db7704e21272433d0f44cf4e6e7e2783bed80f062dcf76748f523f83ca"}
Oct 30 03:30:23.170649 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.170295    4528 scope.go:122] "RemoveContainer" containerID="67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"
Oct 30 03:30:23.178791 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.178635    4528 scope.go:122] "RemoveContainer" containerID="2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"
Oct 30 03:30:23.186014 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.185991    4528 scope.go:122] "RemoveContainer" containerID="2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"
Oct 30 03:30:23.192787 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.192771    4528 scope.go:122] "RemoveContainer" containerID="67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"
Oct 30 03:30:23.193153 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:23.193127    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448\": container with ID starting with 67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448 not found: ID does not exist" containerID="67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"
Oct 30 03:30:23.193220 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.193167    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448"} err="failed to get container status \"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448\": rpc error: code = NotFound desc = could not find container \"67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448\": container with ID starting with 67464fad438d10386cd67737d42dc60832012145e48ee243cdc91ec59fae8448 not found: ID does not exist"
Oct 30 03:30:23.193220 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.193192    4528 scope.go:122] "RemoveContainer" containerID="2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"
Oct 30 03:30:23.193457 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:23.193440    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0\": container with ID starting with 2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0 not found: ID does not exist" containerID="2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"
Oct 30 03:30:23.193494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.193467    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0"} err="failed to get container status \"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0\": rpc error: code = NotFound desc = could not find container \"2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0\": container with ID starting with 2f64b44c6674617a17f0dc580df2c0e4d3b12e13bc21430c1a107ba30eb2ccd0 not found: ID does not exist"
Oct 30 03:30:23.193494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.193487    4528 scope.go:122] "RemoveContainer" containerID="2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"
Oct 30 03:30:23.193724 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:23.193708    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1\": container with ID starting with 2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1 not found: ID does not exist" containerID="2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"
Oct 30 03:30:23.193769 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.193727    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1"} err="failed to get container status \"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1\": rpc error: code = NotFound desc = could not find container \"2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1\": container with ID starting with 2635dfaf30f03e6cf83e81d17562941215938f1f662142f6eef8cec2f0bb35e1 not found: ID does not exist"
Oct 30 03:30:23.227968 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.227940    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:23.238250 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.238224    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:23.892038 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:23.892006    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" path="/var/lib/kubelet/pods/96575aae-7083-46a5-9a3e-fd7864e8f987/volumes"
Oct 30 03:30:25.107695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.107659    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:25.238431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.238390    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:25.363626 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.363535    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:25.449038 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449005    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:30:25.449622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449593    4528 cpu_manager.go:414] "containerMap: removing container" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo"
Oct 30 03:30:25.449622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449618    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449630    4528 cpu_manager.go:414] "containerMap: removing container" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449638    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449648    4528 cpu_manager.go:414] "containerMap: removing container" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway-opa"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449655    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway-opa"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449679    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d4935057-09e7-4c80-b11e-c193928140c5" containerName="otc-container"
Oct 30 03:30:25.449767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449686    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4935057-09e7-4c80-b11e-c193928140c5" containerName="otc-container"
Oct 30 03:30:25.449978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449782    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway-opa"
Oct 30 03:30:25.449978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449797    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4935057-09e7-4c80-b11e-c193928140c5" containerName="otc-container"
Oct 30 03:30:25.449978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449810    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo"
Oct 30 03:30:25.449978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.449834    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="96575aae-7083-46a5-9a3e-fd7864e8f987" containerName="tempo-gateway"
Oct 30 03:30:25.454709 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.454690    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.474537 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.474512    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:30:25.495006 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.494972    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:30:25.498628 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.498583    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.498782 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.498620    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:25.521874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.521844    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:30:25.525023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.524997    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 03:30:25.525023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.525028    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:30:25.525186 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.525073    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-8tg4z\""
Oct 30 03:30:25.525186 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.525118    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-fcf9x\""
Oct 30 03:30:25.530333 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530309    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:30:25.530452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530348    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:30:25.530452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530349    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 03:30:25.530452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530353    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:30:25.530452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530378    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:30:25.530452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.530437    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:30:25.573617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573586    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.573784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573626    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.573784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573667    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.573784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573764    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.573927 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573907    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.573966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573955    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529pl\" (UniqueName: \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.574005 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573973    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.574005 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.573995    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78kn\" (UniqueName: \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.622603 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.622516    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:25.674708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674674    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.674708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674708    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674739    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674772    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529pl\" (UniqueName: \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674788    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674804    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78kn\" (UniqueName: \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674859    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.674980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.674883    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.675539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.675507    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.675640 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.675575    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.675640 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.675595    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.675751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.675661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.677350 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.677324    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.677458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.677395    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.696264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.696239    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78kn\" (UniqueName: \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-6kqls\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.696396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.696298    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-529pl\" (UniqueName: \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-kf4s2\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:25.768891 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.768858    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:30:25.812212 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:25.812182    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:30:26.068039 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.067999    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:30:26.076420 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:30:26.076389    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533931b0_6c31_40ec_9f56_0e130cdae093.slice/crio-7244ecc960c1fb507c0bb2ff3cc15ebb5755267bb3cacdc54fabd5437695559a WatchSource:0}: Error finding container 7244ecc960c1fb507c0bb2ff3cc15ebb5755267bb3cacdc54fabd5437695559a: Status 404 returned error can't find the container with id 7244ecc960c1fb507c0bb2ff3cc15ebb5755267bb3cacdc54fabd5437695559a
Oct 30 03:30:26.090505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.090471    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:26.116731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.116707    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:30:26.121734 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:30:26.121708    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca34685d_7051_44a1_8727_ec91b710bce3.slice/crio-7d6e5767d2e2e6a0feef16b16b4ba33bfbeae08ead2419a1e18bbfbe005192dc WatchSource:0}: Error finding container 7d6e5767d2e2e6a0feef16b16b4ba33bfbeae08ead2419a1e18bbfbe005192dc: Status 404 returned error can't find the container with id 7d6e5767d2e2e6a0feef16b16b4ba33bfbeae08ead2419a1e18bbfbe005192dc
Oct 30 03:30:26.184861 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.184830    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerStarted","Data":"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"}
Oct 30 03:30:26.184985 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.184895    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerStarted","Data":"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"}
Oct 30 03:30:26.184985 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.184922    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerStarted","Data":"7244ecc960c1fb507c0bb2ff3cc15ebb5755267bb3cacdc54fabd5437695559a"}
Oct 30 03:30:26.186380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.186358    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerStarted","Data":"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"}
Oct 30 03:30:26.186476 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.186404    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerStarted","Data":"7d6e5767d2e2e6a0feef16b16b4ba33bfbeae08ead2419a1e18bbfbe005192dc"}
Oct 30 03:30:26.215611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.215577    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:26.224118 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.224074    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" podStartSLOduration=1.22406064 podStartE2EDuration="1.22406064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:26.223706839 +0000 UTC m=+2984.447956116" watchObservedRunningTime="2026-10-30 03:30:26.22406064 +0000 UTC m=+2984.448309907"
Oct 30 03:30:26.353926 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:26.353878    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:27.135039 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.134992    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:30:27.139677 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.139655    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.162702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.162675    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:30:27.169754 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.169697    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:27.185440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.185409    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.185611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.185455    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lqc\" (UniqueName: \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.185611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.185479    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.185611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.185552    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.185775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.185747    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.190174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.190111    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\""
Oct 30 03:30:27.193732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.193706    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerStarted","Data":"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"}
Oct 30 03:30:27.195937 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.195913    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-sx58n\""
Oct 30 03:30:27.196099 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.195995    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:30:27.196099 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.195997    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:30:27.196099 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.196072    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:30:27.231396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.231349    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" podStartSLOduration=2.2313368000000002 podStartE2EDuration="2.2313368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:27.230587055 +0000 UTC m=+2985.454836332" watchObservedRunningTime="2026-10-30 03:30:27.2313368 +0000 UTC m=+2985.455586077"
Oct 30 03:30:27.286514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.286484    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.286686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.286546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.286686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.286584    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lqc\" (UniqueName: \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.286686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.286607    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.286686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.286624    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.287451 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.287423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.289189 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.289165    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.289486 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.289467    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.289560 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.289485    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.307162 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.307129    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:27.312732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.312710    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lqc\" (UniqueName: \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\") pod \"data-science-collector-collector-1\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.441335 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.441230    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:27.454107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.454074    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:30:27.750917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.750836    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:30:27.755855 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:30:27.755813    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc558504b_2aa3_4a27_a3f9_9136903d8a38.slice/crio-a5fedb3390c769734b155c47776f04ec3b1bb352276489ba9825325f46e34207 WatchSource:0}: Error finding container a5fedb3390c769734b155c47776f04ec3b1bb352276489ba9825325f46e34207: Status 404 returned error can't find the container with id a5fedb3390c769734b155c47776f04ec3b1bb352276489ba9825325f46e34207
Oct 30 03:30:27.758394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.758364    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:30:27.758453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:27.758423    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:30:28.199371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.199335    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c558504b-2aa3-4a27-a3f9-9136903d8a38","Type":"ContainerStarted","Data":"d62191d9518f000ee553ffee0741adc0435378a11de9c2f200fd459021d84894"}
Oct 30 03:30:28.199922 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.199393    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c558504b-2aa3-4a27-a3f9-9136903d8a38","Type":"ContainerStarted","Data":"a5fedb3390c769734b155c47776f04ec3b1bb352276489ba9825325f46e34207"}
Oct 30 03:30:28.237846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.237765    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.237745259 podStartE2EDuration="1.237745259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:28.236762196 +0000 UTC m=+2986.461011473" watchObservedRunningTime="2026-10-30 03:30:28.237745259 +0000 UTC m=+2986.461994537"
Oct 30 03:30:28.545869 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.545779    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:28.552116 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.552093    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.567252 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.567225    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:30:28.567252 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.567225    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:30:28.567449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.567297    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:30:28.577082 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.576984    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:30:28.577082 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.577009    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:30:28.577279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.577101    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-bvtwn\""
Oct 30 03:30:28.577279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.577135    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:30:28.577279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.577274    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:30:28.578978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.578958    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:28.596527 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596564    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596665    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596760 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596760 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596750    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7fd\" (UniqueName: \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596857 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596790    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.596857 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.596842    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.597024 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.597006    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.621340 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.621297    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:28.697328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697293    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697337    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697367    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697398    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697416    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:28.697442    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:30:28.697521 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:28.697515    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert podName:960d9109-da32-484c-914e-dc0c668c8ee1 nodeName:}" failed. No retries permitted until 2026-10-30 03:30:29.197492923 +0000 UTC m=+2987.421742198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:30:28.698173 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.698147    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.698294 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.698277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.698336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.698309    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.698372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.697443    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.698421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.698405    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7fd\" (UniqueName: \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.698474 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.698455    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.699102 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.699075    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.700225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.700204    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.700892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.700862    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.720431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.720407    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7fd\" (UniqueName: \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:28.919641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:28.919599    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:29.202501 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:29.202415    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:29.205386 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:29.205358    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:30.794149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:30.794094    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.044143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.044099    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.182030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.181990    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.425996 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.425958    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.547434 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.547397    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:31.574360 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.574327    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:30:31.580218 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.580191    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.595416 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.595384    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:30:31.622260 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622228    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:30:31.622260 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622266    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:30:31.622452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622276    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:30:31.622452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622270    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:30:31.622452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622270    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:30:31.622452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622270    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-4s68n\""
Oct 30 03:30:31.622452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.622325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:30:31.654206 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.654179    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:30:31.654206 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.654196    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:30:31.654410 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.654179    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:30:31.720899 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.720786    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.720915    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.720980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721043    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721098    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721125    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721142    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721168    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721349 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721349 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721349 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721301    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825w6\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.721349 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.721337    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.821990    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822020    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822045    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825w6\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822112    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822210    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822253    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822277    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822309    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822345    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.822569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.822373    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.823145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.823014    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.823145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.823062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.823897 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.823870    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.825177 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825104    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.825716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825407    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.825716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825421    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.825716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825486    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.825716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825667    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.826008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825745    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.826008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.825890    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.826411 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.826391    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:31.843499 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:31.843477    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825w6\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:34.630812 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:30:34.630775    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:30:35.150674 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.150631    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.153003 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.150748    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.175558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.175527    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.175558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.175527    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.251493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.251459    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.251699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.251567    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.251699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.251623    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.251699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.251666    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.264274 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.264242    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") DevicePath \"csi-e8943ccbf9e44c5d22a63e7297537a58134696922d8a2a16f63173089ab7966e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.264387 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.264334    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.364266 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.364232    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.364465 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.364436    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.389156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.389125    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.391396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.391354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.467493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.467408    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:35.500473 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.500437    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:35.731617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.731543    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:30:35.797658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.797623    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:30:35.802205 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:30:35.802175    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960d9109_da32_484c_914e_dc0c668c8ee1.slice/crio-8fb76590e6bbdc8c440b1808b32d44f77471d4007ef6efb0c4a82fe634b8c018 WatchSource:0}: Error finding container 8fb76590e6bbdc8c440b1808b32d44f77471d4007ef6efb0c4a82fe634b8c018: Status 404 returned error can't find the container with id 8fb76590e6bbdc8c440b1808b32d44f77471d4007ef6efb0c4a82fe634b8c018
Oct 30 03:30:35.828313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:35.828255    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:30:35.833648 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:30:35.833624    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b402fa1_86c3_490c_8635_a16e3fd9aeb7.slice/crio-90f11148af6c5ada7043f38c8d4ba24091b34a1a9be647ebe116bbc5e2b4fa8c WatchSource:0}: Error finding container 90f11148af6c5ada7043f38c8d4ba24091b34a1a9be647ebe116bbc5e2b4fa8c: Status 404 returned error can't find the container with id 90f11148af6c5ada7043f38c8d4ba24091b34a1a9be647ebe116bbc5e2b4fa8c
Oct 30 03:30:36.236162 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.236124    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerStarted","Data":"90f11148af6c5ada7043f38c8d4ba24091b34a1a9be647ebe116bbc5e2b4fa8c"}
Oct 30 03:30:36.238265 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.238239    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerStarted","Data":"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"}
Oct 30 03:30:36.238351 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.238295    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerStarted","Data":"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"}
Oct 30 03:30:36.238351 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.238315    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerStarted","Data":"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"}
Oct 30 03:30:36.238351 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.238330    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerStarted","Data":"8fb76590e6bbdc8c440b1808b32d44f77471d4007ef6efb0c4a82fe634b8c018"}
Oct 30 03:30:36.238460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.238355    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:36.240780 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.240760    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:36.281079 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:36.280977    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=8.280962266 podStartE2EDuration="8.280962266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:36.280063639 +0000 UTC m=+2994.504312942" watchObservedRunningTime="2026-10-30 03:30:36.280962266 +0000 UTC m=+2994.505211542"
Oct 30 03:30:37.244012 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:37.243972    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerStarted","Data":"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"}
Oct 30 03:30:37.244464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:37.244089    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:37.244464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:37.244139    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:37.251866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:37.251834    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:30:40.260781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:40.260741    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84" exitCode=0
Oct 30 03:30:40.261208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:40.260831    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerDied","Data":"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"}
Oct 30 03:30:41.267517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:41.267484    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerStarted","Data":"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"}
Oct 30 03:30:41.267925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:41.267548    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerStarted","Data":"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"}
Oct 30 03:30:41.267925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:41.267573    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerStarted","Data":"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"}
Oct 30 03:30:41.343382 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:41.343330    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.955647027 podStartE2EDuration="10.343316739s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.387669712s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:41.332794781 +0000 UTC m=+2999.557044058" watchObservedRunningTime="2026-10-30 03:30:41.343316739 +0000 UTC m=+2999.567566057"
Oct 30 03:30:45.500957 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:45.500914    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:50.501310 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:50.501274    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:50.506858 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:50.506835    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:51.314189 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:51.314161    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:30:59.254672 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:30:59.254571    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:31:04.631649 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:31:04.631611    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:31:09.253733 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:31:09.253689    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:31:19.254643 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:31:19.254607    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:31:34.527612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:31:34.527574    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:34.630656 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:31:34.630623    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:31:34.650604 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:31:34.650569    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:31:35.849949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:31:35.849910    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:32:04.632016 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:32:04.631977    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:32:34.629864 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:32:34.629806    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:32:35.980987 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:32:35.980948    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:04.631891 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:33:04.631852    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:33:34.631215 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:33:34.631168    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:33:36.057887 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:33:36.057859    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:33:53.912333 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:33:53.912247    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:04.630488 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:34:04.630451    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:34:34.631417 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:34:34.631381    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:34:36.228581 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:34:36.228544    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:34:37.676235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:34:37.676197    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:35:04.631198 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:35:04.631157    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:35:34.633365 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:35:34.633322    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:35:36.348620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:35:36.348565    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:36:04.631454 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:36:04.631417    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:36:34.631226 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:36:34.631176    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:36:36.495029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:36:36.494994    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:04.631685 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:37:04.631601    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:37:34.631312 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:37:34.631274    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:37:36.609559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:37:36.609523    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:52.967091 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:37:52.967049    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:37:54.393094 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:37:54.393051    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:38:04.631605 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:38:04.631566    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:38:34.631259 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:38:34.631221    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:38:36.721721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:38:36.721684    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:39:04.630831 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:39:04.630772    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:39:34.631003 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:39:34.630968    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:39:36.869371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:39:36.869333    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:40:04.632707 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:40:04.632668    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:40:34.632307 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:40:34.632270    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:40:36.968323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:40:36.968290    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:02.097271 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:02.097233    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:02.235016 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:02.234982    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:04.631993 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:04.631959    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:41:21.402235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.402193    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:41:21.402745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.402458    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="kube-rbac-proxy" containerID="cri-o://cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9" gracePeriod=30
Oct 30 03:41:21.402745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.402538    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="prom-label-proxy" containerID="cri-o://fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d" gracePeriod=30
Oct 30 03:41:21.426159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.426136    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:41:21.426383 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.426349    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="kube-rbac-proxy" containerID="cri-o://7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57" gracePeriod=30
Oct 30 03:41:21.426456 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.426427    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="prom-label-proxy" containerID="cri-o://843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0" gracePeriod=30
Oct 30 03:41:21.691190 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.691165    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:41:21.693773 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.693757    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:41:21.736608 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.736575    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\") pod \"533931b0-6c31-40ec-9f56-0e130cdae093\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") "
Oct 30 03:41:21.736790 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.736661    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\") pod \"533931b0-6c31-40ec-9f56-0e130cdae093\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") "
Oct 30 03:41:21.737193 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737168    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\" (UniqueName: \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\") pod \"ca34685d-7051-44a1-8727-ec91b710bce3\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") "
Oct 30 03:41:21.737313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737242    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\") pod \"533931b0-6c31-40ec-9f56-0e130cdae093\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") "
Oct 30 03:41:21.737313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737266    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\" (UniqueName: \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\") pod \"ca34685d-7051-44a1-8727-ec91b710bce3\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") "
Oct 30 03:41:21.737313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737290    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\" (UniqueName: \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\") pod \"ca34685d-7051-44a1-8727-ec91b710bce3\" (UID: \"ca34685d-7051-44a1-8727-ec91b710bce3\") "
Oct 30 03:41:21.737475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737329    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\" (UniqueName: \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\") pod \"533931b0-6c31-40ec-9f56-0e130cdae093\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") "
Oct 30 03:41:21.737475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.737344    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\" (UniqueName: \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\") pod \"533931b0-6c31-40ec-9f56-0e130cdae093\" (UID: \"533931b0-6c31-40ec-9f56-0e130cdae093\") "
Oct 30 03:41:21.738362 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.738313    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config" pod "533931b0-6c31-40ec-9f56-0e130cdae093" (UID: "533931b0-6c31-40ec-9f56-0e130cdae093"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:21.738362 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.738349    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config" pod "ca34685d-7051-44a1-8727-ec91b710bce3" (UID: "ca34685d-7051-44a1-8727-ec91b710bce3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:21.738964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.738863    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca" pod "533931b0-6c31-40ec-9f56-0e130cdae093" (UID: "533931b0-6c31-40ec-9f56-0e130cdae093"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:21.739198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.739107    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca" pod "533931b0-6c31-40ec-9f56-0e130cdae093" (UID: "533931b0-6c31-40ec-9f56-0e130cdae093"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:21.741519 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.741476    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn" pod "533931b0-6c31-40ec-9f56-0e130cdae093" (UID: "533931b0-6c31-40ec-9f56-0e130cdae093"). InnerVolumeSpecName "kube-api-access-v78kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:21.741741 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.741682    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl" pod "ca34685d-7051-44a1-8727-ec91b710bce3" (UID: "ca34685d-7051-44a1-8727-ec91b710bce3"). InnerVolumeSpecName "kube-api-access-529pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:21.742405 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.742383    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls" pod "ca34685d-7051-44a1-8727-ec91b710bce3" (UID: "ca34685d-7051-44a1-8727-ec91b710bce3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:21.743445 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.743421    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls" pod "533931b0-6c31-40ec-9f56-0e130cdae093" (UID: "533931b0-6c31-40ec-9f56-0e130cdae093"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:21.822296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.822262    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:21.838562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838532    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ca34685d-7051-44a1-8727-ec91b710bce3-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838558    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v78kn\" (UniqueName: \"kubernetes.io/projected/533931b0-6c31-40ec-9f56-0e130cdae093-kube-api-access-v78kn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838568    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/533931b0-6c31-40ec-9f56-0e130cdae093-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838577    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-prometheus-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838589    4528 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838597    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca34685d-7051-44a1-8727-ec91b710bce3-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838609    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/533931b0-6c31-40ec-9f56-0e130cdae093-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.838863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.838622    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-529pl\" (UniqueName: \"kubernetes.io/projected/ca34685d-7051-44a1-8727-ec91b710bce3-kube-api-access-529pl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:21.894622 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.894591    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:41:21.895125 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.894915    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="prometheus" containerID="cri-o://4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607" gracePeriod=600
Oct 30 03:41:21.895125 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.894968    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="thanos-sidecar" containerID="cri-o://b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db" gracePeriod=600
Oct 30 03:41:21.895125 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.895093    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="config-reloader" containerID="cri-o://918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871" gracePeriod=600
Oct 30 03:41:21.946051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946013    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:21.946051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946035    4528 generic.go:356] "Generic (PLEG): container finished" podID="533931b0-6c31-40ec-9f56-0e130cdae093" containerID="fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d" exitCode=0
Oct 30 03:41:21.946222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946061    4528 generic.go:356] "Generic (PLEG): container finished" podID="533931b0-6c31-40ec-9f56-0e130cdae093" containerID="cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9" exitCode=0
Oct 30 03:41:21.946222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946184    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerDied","Data":"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"}
Oct 30 03:41:21.946307 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946286    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerDied","Data":"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"}
Oct 30 03:41:21.947421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls" event={"ID":"533931b0-6c31-40ec-9f56-0e130cdae093","Type":"ContainerDied","Data":"7244ecc960c1fb507c0bb2ff3cc15ebb5755267bb3cacdc54fabd5437695559a"}
Oct 30 03:41:21.947421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946363    4528 scope.go:122] "RemoveContainer" containerID="fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"
Oct 30 03:41:21.947421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.946989    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"
Oct 30 03:41:21.948973 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.948950    4528 generic.go:356] "Generic (PLEG): container finished" podID="ca34685d-7051-44a1-8727-ec91b710bce3" containerID="843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0" exitCode=0
Oct 30 03:41:21.948973 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.948973    4528 generic.go:356] "Generic (PLEG): container finished" podID="ca34685d-7051-44a1-8727-ec91b710bce3" containerID="7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57" exitCode=0
Oct 30 03:41:21.949319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.949004    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerDied","Data":"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"}
Oct 30 03:41:21.949319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.949033    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"
Oct 30 03:41:21.949319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.949046    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerDied","Data":"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"}
Oct 30 03:41:21.949319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:21.949073    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2" event={"ID":"ca34685d-7051-44a1-8727-ec91b710bce3","Type":"ContainerDied","Data":"7d6e5767d2e2e6a0feef16b16b4ba33bfbeae08ead2419a1e18bbfbe005192dc"}
Oct 30 03:41:22.008703 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.008676    4528 scope.go:122] "RemoveContainer" containerID="cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"
Oct 30 03:41:22.014424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.014394    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:41:22.018621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.018134    4528 scope.go:122] "RemoveContainer" containerID="fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"
Oct 30 03:41:22.018621 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:22.018469    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d\": container with ID starting with fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d not found: ID does not exist" containerID="fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"
Oct 30 03:41:22.018621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.018506    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"} err="failed to get container status \"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d\": rpc error: code = NotFound desc = could not find container \"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d\": container with ID starting with fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d not found: ID does not exist"
Oct 30 03:41:22.018621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.018533    4528 scope.go:122] "RemoveContainer" containerID="cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"
Oct 30 03:41:22.018998 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:22.018794    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9\": container with ID starting with cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9 not found: ID does not exist" containerID="cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"
Oct 30 03:41:22.018998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.018874    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"} err="failed to get container status \"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9\": rpc error: code = NotFound desc = could not find container \"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9\": container with ID starting with cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9 not found: ID does not exist"
Oct 30 03:41:22.018998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.018901    4528 scope.go:122] "RemoveContainer" containerID="fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"
Oct 30 03:41:22.019574 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.019363    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d"} err="failed to get container status \"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d\": rpc error: code = NotFound desc = could not find container \"fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d\": container with ID starting with fa0eaa11192069a535d9861584bb37c6eb6d9d81f0cc7a63faae18ec1901405d not found: ID does not exist"
Oct 30 03:41:22.019574 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.019390    4528 scope.go:122] "RemoveContainer" containerID="cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"
Oct 30 03:41:22.020377 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.019789    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9"} err="failed to get container status \"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9\": rpc error: code = NotFound desc = could not find container \"cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9\": container with ID starting with cdbf851f78373b407c7bf4c3eca8e30ef918faae311efedf1b3ddb7035d246c9 not found: ID does not exist"
Oct 30 03:41:22.020377 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.019844    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-6kqls"]
Oct 30 03:41:22.020377 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.019811    4528 scope.go:122] "RemoveContainer" containerID="843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"
Oct 30 03:41:22.028404 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.028323    4528 scope.go:122] "RemoveContainer" containerID="7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"
Oct 30 03:41:22.037751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.037729    4528 scope.go:122] "RemoveContainer" containerID="843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"
Oct 30 03:41:22.038178 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:22.038157    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0\": container with ID starting with 843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0 not found: ID does not exist" containerID="843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"
Oct 30 03:41:22.038268 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038185    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"} err="failed to get container status \"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0\": rpc error: code = NotFound desc = could not find container \"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0\": container with ID starting with 843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0 not found: ID does not exist"
Oct 30 03:41:22.038268 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038203    4528 scope.go:122] "RemoveContainer" containerID="7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"
Oct 30 03:41:22.038464 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:22.038443    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57\": container with ID starting with 7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57 not found: ID does not exist" containerID="7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"
Oct 30 03:41:22.038545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038470    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"} err="failed to get container status \"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57\": rpc error: code = NotFound desc = could not find container \"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57\": container with ID starting with 7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57 not found: ID does not exist"
Oct 30 03:41:22.038545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038487    4528 scope.go:122] "RemoveContainer" containerID="843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"
Oct 30 03:41:22.038859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038811    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0"} err="failed to get container status \"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0\": rpc error: code = NotFound desc = could not find container \"843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0\": container with ID starting with 843cd708f27ea99552cab0650e82f0e18c6c5d484de9de2217e2362f4dad8bf0 not found: ID does not exist"
Oct 30 03:41:22.038952 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.038860    4528 scope.go:122] "RemoveContainer" containerID="7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"
Oct 30 03:41:22.039190 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.039161    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57"} err="failed to get container status \"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57\": rpc error: code = NotFound desc = could not find container \"7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57\": container with ID starting with 7230ac1e76500ee6c0239b1ef155464325f2ffcdef0f3051b38df701609ecc57 not found: ID does not exist"
Oct 30 03:41:22.050473 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.050451    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:41:22.055337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.055284    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-kf4s2"]
Oct 30 03:41:22.056245 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.056207    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:22.141353 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.141328    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:41:22.241147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241065    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241143    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241162    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241189    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241247    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241288    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241313    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241383    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.241782 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.241589    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:22.242331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242168    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:22.242331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242288    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.242655 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242394    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.242655 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242540    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\" (UID: \"9b402fa1-86c3-490c-8635-a16e3fd9aeb7\") "
Oct 30 03:41:22.242896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242846    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:22.242984 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242932    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.242984 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242951    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.242984 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.242969    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.244308 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.244282    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.244404 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.244381    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:41:22.245341 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.245312    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:22.246139 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.246102    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.246235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.246187    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "kube-api-access-825w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:22.246704 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.246662    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.246807 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.246725    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.247166 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.247088    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.252438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.252413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:41:22.260865 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.260813    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config" pod "9b402fa1-86c3-490c-8635-a16e3fd9aeb7" (UID: "9b402fa1-86c3-490c-8635-a16e3fd9aeb7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:22.304543 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.304517    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:22.343544 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343518    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343544 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343540    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343552    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343562    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343572    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343582    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343590    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343616    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343625    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825w6\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-kube-api-access-825w6\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.343717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.343633    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b402fa1-86c3-490c-8635-a16e3fd9aeb7-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:22.354466 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.354449    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:41:22.416208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.414888    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:22.444738 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.444707    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-e8943ccbf9e44c5d22a63e7297537a58134696922d8a2a16f63173089ab7966e\""
Oct 30 03:41:22.764851 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.763657    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:22.764851 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.763948    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c558504b-2aa3-4a27-a3f9-9136903d8a38" containerName="otc-container" containerID="cri-o://d62191d9518f000ee553ffee0741adc0435378a11de9c2f200fd459021d84894" gracePeriod=30
Oct 30 03:41:22.803970 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.803938    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:22.928273 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.928237    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:22.956695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956662    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db" exitCode=0
Oct 30 03:41:22.956695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956691    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871" exitCode=0
Oct 30 03:41:22.956695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956697    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607" exitCode=0
Oct 30 03:41:22.957002 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956737    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerDied","Data":"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"}
Oct 30 03:41:22.957002 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956746    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 03:41:22.958283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956805    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerDied","Data":"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"}
Oct 30 03:41:22.958283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.957779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerDied","Data":"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"}
Oct 30 03:41:22.958283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.957867    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"9b402fa1-86c3-490c-8635-a16e3fd9aeb7","Type":"ContainerDied","Data":"90f11148af6c5ada7043f38c8d4ba24091b34a1a9be647ebe116bbc5e2b4fa8c"}
Oct 30 03:41:22.958283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.956837    4528 scope.go:122] "RemoveContainer" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"
Oct 30 03:41:22.958921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.958899    4528 generic.go:356] "Generic (PLEG): container finished" podID="c558504b-2aa3-4a27-a3f9-9136903d8a38" containerID="d62191d9518f000ee553ffee0741adc0435378a11de9c2f200fd459021d84894" exitCode=0
Oct 30 03:41:22.958991 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.958939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c558504b-2aa3-4a27-a3f9-9136903d8a38","Type":"ContainerDied","Data":"d62191d9518f000ee553ffee0741adc0435378a11de9c2f200fd459021d84894"}
Oct 30 03:41:22.967084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.966908    4528 scope.go:122] "RemoveContainer" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"
Oct 30 03:41:22.976675 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.976652    4528 scope.go:122] "RemoveContainer" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"
Oct 30 03:41:22.989975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.989945    4528 scope.go:122] "RemoveContainer" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"
Oct 30 03:41:22.999896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:22.999872    4528 scope.go:122] "RemoveContainer" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"
Oct 30 03:41:23.000202 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:23.000176    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": container with ID starting with b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db not found: ID does not exist" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"
Oct 30 03:41:23.000273 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000209    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"} err="failed to get container status \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": rpc error: code = NotFound desc = could not find container \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": container with ID starting with b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db not found: ID does not exist"
Oct 30 03:41:23.000273 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000229    4528 scope.go:122] "RemoveContainer" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"
Oct 30 03:41:23.000423 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:23.000408    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": container with ID starting with 918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871 not found: ID does not exist" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"
Oct 30 03:41:23.000465 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000429    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"} err="failed to get container status \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": rpc error: code = NotFound desc = could not find container \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": container with ID starting with 918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871 not found: ID does not exist"
Oct 30 03:41:23.000465 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000446    4528 scope.go:122] "RemoveContainer" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"
Oct 30 03:41:23.000717 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:23.000697    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": container with ID starting with 4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607 not found: ID does not exist" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"
Oct 30 03:41:23.000785 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000723    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"} err="failed to get container status \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": rpc error: code = NotFound desc = could not find container \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": container with ID starting with 4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607 not found: ID does not exist"
Oct 30 03:41:23.000785 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.000738    4528 scope.go:122] "RemoveContainer" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"
Oct 30 03:41:23.001071 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:23.001021    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": container with ID starting with 9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84 not found: ID does not exist" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"
Oct 30 03:41:23.001071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001041    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"} err="failed to get container status \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": rpc error: code = NotFound desc = could not find container \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": container with ID starting with 9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84 not found: ID does not exist"
Oct 30 03:41:23.001071 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001054    4528 scope.go:122] "RemoveContainer" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"
Oct 30 03:41:23.001296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001280    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"} err="failed to get container status \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": rpc error: code = NotFound desc = could not find container \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": container with ID starting with b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db not found: ID does not exist"
Oct 30 03:41:23.001296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001295    4528 scope.go:122] "RemoveContainer" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"
Oct 30 03:41:23.001519 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001501    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"} err="failed to get container status \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": rpc error: code = NotFound desc = could not find container \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": container with ID starting with 918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871 not found: ID does not exist"
Oct 30 03:41:23.001519 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001517    4528 scope.go:122] "RemoveContainer" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"
Oct 30 03:41:23.001714 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001698    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"} err="failed to get container status \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": rpc error: code = NotFound desc = could not find container \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": container with ID starting with 4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607 not found: ID does not exist"
Oct 30 03:41:23.001772 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001714    4528 scope.go:122] "RemoveContainer" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"
Oct 30 03:41:23.001940 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001923    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"} err="failed to get container status \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": rpc error: code = NotFound desc = could not find container \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": container with ID starting with 9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84 not found: ID does not exist"
Oct 30 03:41:23.001940 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.001939    4528 scope.go:122] "RemoveContainer" containerID="b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"
Oct 30 03:41:23.002131 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002112    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db"} err="failed to get container status \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": rpc error: code = NotFound desc = could not find container \"b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db\": container with ID starting with b2c41772144d21b05e2fe3d6514e8e6325a0627985ed5159f5de7283197fd7db not found: ID does not exist"
Oct 30 03:41:23.002189 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002130    4528 scope.go:122] "RemoveContainer" containerID="918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"
Oct 30 03:41:23.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002299    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871"} err="failed to get container status \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": rpc error: code = NotFound desc = could not find container \"918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871\": container with ID starting with 918382ac2e28f3b0db52b016cd09a3cbc73c60052b3d47b2dff202c1b71f3871 not found: ID does not exist"
Oct 30 03:41:23.002318 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002317    4528 scope.go:122] "RemoveContainer" containerID="4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"
Oct 30 03:41:23.002513 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002493    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607"} err="failed to get container status \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": rpc error: code = NotFound desc = could not find container \"4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607\": container with ID starting with 4ac286aba01e54d5f108fcdf6e8e496badc2b2019bad100b5d60af4aa5a56607 not found: ID does not exist"
Oct 30 03:41:23.002579 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002514    4528 scope.go:122] "RemoveContainer" containerID="9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"
Oct 30 03:41:23.002719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.002699    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84"} err="failed to get container status \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": rpc error: code = NotFound desc = could not find container \"9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84\": container with ID starting with 9715749f4ab728d4c352d305787f5a8308add93443d3031ae78cf3c5f75bbf84 not found: ID does not exist"
Oct 30 03:41:23.010719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.010697    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:41:23.020863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.020842    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 03:41:23.032518 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.032500    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:23.046641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.046613    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:23.151250 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151215    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\") pod \"c558504b-2aa3-4a27-a3f9-9136903d8a38\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") "
Oct 30 03:41:23.151419 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151289    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\") pod \"c558504b-2aa3-4a27-a3f9-9136903d8a38\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") "
Oct 30 03:41:23.151419 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151327    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\") pod \"c558504b-2aa3-4a27-a3f9-9136903d8a38\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") "
Oct 30 03:41:23.151419 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151368    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\") pod \"c558504b-2aa3-4a27-a3f9-9136903d8a38\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") "
Oct 30 03:41:23.151585 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\" (UniqueName: \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\") pod \"c558504b-2aa3-4a27-a3f9-9136903d8a38\" (UID: \"c558504b-2aa3-4a27-a3f9-9136903d8a38\") "
Oct 30 03:41:23.151708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal" pod "c558504b-2aa3-4a27-a3f9-9136903d8a38" (UID: "c558504b-2aa3-4a27-a3f9-9136903d8a38"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:23.151863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.151772    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c558504b-2aa3-4a27-a3f9-9136903d8a38-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:23.153530 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.153508    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs" pod "c558504b-2aa3-4a27-a3f9-9136903d8a38" (UID: "c558504b-2aa3-4a27-a3f9-9136903d8a38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:23.153629 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.153564    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc" pod "c558504b-2aa3-4a27-a3f9-9136903d8a38" (UID: "c558504b-2aa3-4a27-a3f9-9136903d8a38"). InnerVolumeSpecName "kube-api-access-v2lqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:23.153813 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.153793    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert" pod "c558504b-2aa3-4a27-a3f9-9136903d8a38" (UID: "c558504b-2aa3-4a27-a3f9-9136903d8a38"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:23.153922 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.153848    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs" pod "c558504b-2aa3-4a27-a3f9-9136903d8a38" (UID: "c558504b-2aa3-4a27-a3f9-9136903d8a38"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:23.252365 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.252330    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:23.252365 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.252361    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:23.252365 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.252371    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2lqc\" (UniqueName: \"kubernetes.io/projected/c558504b-2aa3-4a27-a3f9-9136903d8a38-kube-api-access-v2lqc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:23.252583 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.252380    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c558504b-2aa3-4a27-a3f9-9136903d8a38-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:23.622639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.622600    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:23.623130 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.622885    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo" containerID="cri-o://313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33" gracePeriod=30
Oct 30 03:41:23.623272 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.623249    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway-opa" containerID="cri-o://fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969" gracePeriod=30
Oct 30 03:41:23.623354 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.623340    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway" containerID="cri-o://05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c" gracePeriod=30
Oct 30 03:41:23.690679 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.690648    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:23.894078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.894040    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" path="/var/lib/kubelet/pods/533931b0-6c31-40ec-9f56-0e130cdae093/volumes"
Oct 30 03:41:23.894739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.894719    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" path="/var/lib/kubelet/pods/9b402fa1-86c3-490c-8635-a16e3fd9aeb7/volumes"
Oct 30 03:41:23.895581 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.895559    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" path="/var/lib/kubelet/pods/ca34685d-7051-44a1-8727-ec91b710bce3/volumes"
Oct 30 03:41:23.963731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.963695    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:23.968062 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.968032    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c558504b-2aa3-4a27-a3f9-9136903d8a38","Type":"ContainerDied","Data":"a5fedb3390c769734b155c47776f04ec3b1bb352276489ba9825325f46e34207"}
Oct 30 03:41:23.968228 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.968049    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:23.968228 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.968097    4528 scope.go:122] "RemoveContainer" containerID="d62191d9518f000ee553ffee0741adc0435378a11de9c2f200fd459021d84894"
Oct 30 03:41:23.970632 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.970611    4528 generic.go:356] "Generic (PLEG): container finished" podID="960d9109-da32-484c-914e-dc0c668c8ee1" containerID="fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969" exitCode=0
Oct 30 03:41:23.970632 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.970632    4528 generic.go:356] "Generic (PLEG): container finished" podID="960d9109-da32-484c-914e-dc0c668c8ee1" containerID="313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33" exitCode=0
Oct 30 03:41:23.970806 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.970671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerDied","Data":"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"}
Oct 30 03:41:23.970806 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:23.970708    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerDied","Data":"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"}
Oct 30 03:41:24.012854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.012807    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:24.017489 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.017462    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:24.074727 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.074690    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:24.196741 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.196655    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:24.864398 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.864373    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:24.969938 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.969904    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.969938 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.969945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\" (UniqueName: \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.969974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970009    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970113    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970161    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970222    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970251    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970287    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\") pod \"960d9109-da32-484c-914e-dc0c668c8ee1\" (UID: \"960d9109-da32-484c-914e-dc0c668c8ee1\") "
Oct 30 03:41:24.970489 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970365    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:24.971001 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970640    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:24.971001 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970805    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:24.971001 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.970810    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:24.972662 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.972541    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:24.972775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.972680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:24.972775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.972720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:24.972960 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.972833    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "kube-api-access-pv7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:24.972960 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.972947    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:24.973620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.973596    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "960d9109-da32-484c-914e-dc0c668c8ee1" (UID: "960d9109-da32-484c-914e-dc0c668c8ee1"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:41:24.978705 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.978679    4528 generic.go:356] "Generic (PLEG): container finished" podID="960d9109-da32-484c-914e-dc0c668c8ee1" containerID="05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c" exitCode=0
Oct 30 03:41:24.978828 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.978736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerDied","Data":"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"}
Oct 30 03:41:24.978828 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.978756    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:41:24.978828 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.978790    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"960d9109-da32-484c-914e-dc0c668c8ee1","Type":"ContainerDied","Data":"8fb76590e6bbdc8c440b1808b32d44f77471d4007ef6efb0c4a82fe634b8c018"}
Oct 30 03:41:24.979017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.978834    4528 scope.go:122] "RemoveContainer" containerID="fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"
Oct 30 03:41:24.989761 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.989733    4528 scope.go:122] "RemoveContainer" containerID="05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"
Oct 30 03:41:24.997288 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:24.997270    4528 scope.go:122] "RemoveContainer" containerID="313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"
Oct 30 03:41:25.004711 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.004693    4528 scope.go:122] "RemoveContainer" containerID="fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"
Oct 30 03:41:25.005023 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:25.005005    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969\": container with ID starting with fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969 not found: ID does not exist" containerID="fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"
Oct 30 03:41:25.005091 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.005031    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969"} err="failed to get container status \"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969\": rpc error: code = NotFound desc = could not find container \"fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969\": container with ID starting with fdfd71243169102953aad9c32cbb8a4570d875af25e5ec3673764d1e5a624969 not found: ID does not exist"
Oct 30 03:41:25.005091 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.005047    4528 scope.go:122] "RemoveContainer" containerID="05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"
Oct 30 03:41:25.005310 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:25.005294    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c\": container with ID starting with 05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c not found: ID does not exist" containerID="05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"
Oct 30 03:41:25.005359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.005313    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c"} err="failed to get container status \"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c\": rpc error: code = NotFound desc = could not find container \"05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c\": container with ID starting with 05bfba8f10a299491f52967485841f183d1e190f4ded4ffd16241e7c6f0b9c8c not found: ID does not exist"
Oct 30 03:41:25.005359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.005327    4528 scope.go:122] "RemoveContainer" containerID="313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"
Oct 30 03:41:25.005520 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:25.005505    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33\": container with ID starting with 313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33 not found: ID does not exist" containerID="313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"
Oct 30 03:41:25.005555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.005523    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33"} err="failed to get container status \"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33\": rpc error: code = NotFound desc = could not find container \"313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33\": container with ID starting with 313e13686394becbb272e8f0fdafd809a6a96bdccda3204ee9894725c17a7e33 not found: ID does not exist"
Oct 30 03:41:25.023719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.023661    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:25.027775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.027752    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:41:25.071752 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071713    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071752 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071746    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071752 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071757    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071770    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071781    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv7fd\" (UniqueName: \"kubernetes.io/projected/960d9109-da32-484c-914e-dc0c668c8ee1-kube-api-access-pv7fd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071813    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071841    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/960d9109-da32-484c-914e-dc0c668c8ee1-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:41:25.071982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.071868    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:41:25.080092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.080075    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:41:25.172717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.172684    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:41:25.891927 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.891886    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" path="/var/lib/kubelet/pods/960d9109-da32-484c-914e-dc0c668c8ee1/volumes"
Oct 30 03:41:25.892537 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:25.892514    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c558504b-2aa3-4a27-a3f9-9136903d8a38" path="/var/lib/kubelet/pods/c558504b-2aa3-4a27-a3f9-9136903d8a38/volumes"
Oct 30 03:41:28.609777 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.609740    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:28.724526 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.724492    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:28.841097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.841059    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:28.967598 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.967560    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:28.987032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.986998    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:41:28.987276 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987264    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="prometheus"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987278    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="prometheus"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987289    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="init-config-reloader"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987294    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="init-config-reloader"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987307    4528 cpu_manager.go:414] "containerMap: removing container" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987313    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987319    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c558504b-2aa3-4a27-a3f9-9136903d8a38" containerName="otc-container"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987323    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c558504b-2aa3-4a27-a3f9-9136903d8a38" containerName="otc-container"
Oct 30 03:41:28.987329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987329    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="thanos-sidecar"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987334    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="thanos-sidecar"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987342    4528 cpu_manager.go:414] "containerMap: removing container" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway-opa"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987346    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway-opa"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987351    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="config-reloader"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987357    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="config-reloader"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987371    4528 cpu_manager.go:414] "containerMap: removing container" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987376    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987386    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987391    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987398    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987403    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987408    4528 cpu_manager.go:414] "containerMap: removing container" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987413    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987418    4528 cpu_manager.go:414] "containerMap: removing container" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987423    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987463    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987470    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987475    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca34685d-7051-44a1-8727-ec91b710bce3" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987481    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="prom-label-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987486    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987491    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="prometheus"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987497    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="533931b0-6c31-40ec-9f56-0e130cdae093" containerName="kube-rbac-proxy"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987502    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c558504b-2aa3-4a27-a3f9-9136903d8a38" containerName="otc-container"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987506    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="thanos-sidecar"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987512    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="960d9109-da32-484c-914e-dc0c668c8ee1" containerName="tempo-gateway-opa"
Oct 30 03:41:28.987559 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.987517    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b402fa1-86c3-490c-8635-a16e3fd9aeb7" containerName="config-reloader"
Oct 30 03:41:28.992347 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.992327    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:28.999254 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:28.999227    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:41:29.002958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.002938    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.007920 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.007895    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:41:29.023948 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.023919    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:41:29.094206 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094170    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 03:41:29.094411 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094390    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:41:29.094488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094436    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 03:41:29.094488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094446    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 03:41:29.094488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094453    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 03:41:29.094488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094472    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 03:41:29.094488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094479    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-bz87t\""
Oct 30 03:41:29.094725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094509    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:41:29.094725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094517    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-5v2xc\""
Oct 30 03:41:29.094725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.094551    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:41:29.103983 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.103954    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr2t\" (UniqueName: \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.104080 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104010    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.104132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104109    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.104172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104160    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.104208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104179    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.104208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npx5\" (UniqueName: \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.104312 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.104382 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.104362    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.154254 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.154223    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:29.205025 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.204997    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr2t\" (UniqueName: \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205040    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.205145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205058    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205085    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205113    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205145 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205142    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npx5\" (UniqueName: \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.205391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205202    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.205863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205800    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.205863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205803    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.206036 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205802    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.206036 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.205944    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.207563 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.207544    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.207629 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.207577    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.226047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.225987    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr2t\" (UniqueName: \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-4bc5b\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.226047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.226038    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npx5\" (UniqueName: \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-pmv9m\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.309930 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.309898    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:41:29.320745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.320711    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:41:29.576700 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.576619    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:29.613686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.613653    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:41:29.618986 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:41:29.618958    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeedf29_24be_4352_ac34_e1bb63cf8dc7.slice/crio-2e0fb225b796b64ddd3348ae83dd112fe14c17ace446b3d56db7d4d46983ad3f WatchSource:0}: Error finding container 2e0fb225b796b64ddd3348ae83dd112fe14c17ace446b3d56db7d4d46983ad3f: Status 404 returned error can't find the container with id 2e0fb225b796b64ddd3348ae83dd112fe14c17ace446b3d56db7d4d46983ad3f
Oct 30 03:41:29.642955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.642930    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:41:29.647932 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:41:29.647906    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b687f0b_239a_4fad_95ee_51a0af2ea98a.slice/crio-ea955242348ffa13af476b5625ade70b7f124623182ee2d784a4c1eaf3773ebd WatchSource:0}: Error finding container ea955242348ffa13af476b5625ade70b7f124623182ee2d784a4c1eaf3773ebd: Status 404 returned error can't find the container with id ea955242348ffa13af476b5625ade70b7f124623182ee2d784a4c1eaf3773ebd
Oct 30 03:41:29.759366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.759324    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:29.989493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:29.989455    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:30.000297 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.000263    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerStarted","Data":"ca9d6b3dc4f490e811e14e83c293a54994663056d365c1adc2d8ff3d73831328"}
Oct 30 03:41:30.000454 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.000320    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerStarted","Data":"2ab2e718fcf3c6cbc92b5e9d9707f343bf5ef553445a22786a50e19fd4216d0b"}
Oct 30 03:41:30.000454 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.000353    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerStarted","Data":"2e0fb225b796b64ddd3348ae83dd112fe14c17ace446b3d56db7d4d46983ad3f"}
Oct 30 03:41:30.002109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.002084    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerStarted","Data":"e5a7a90ae9bbe6f486af773849bbf3c7dc48c634752c0dc47eff79177957418d"}
Oct 30 03:41:30.002278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.002130    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerStarted","Data":"ccec5e1207bae82737f2edb16f5e0e40df987b33ae243d93791b220c90407bb6"}
Oct 30 03:41:30.002278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.002150    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerStarted","Data":"ea955242348ffa13af476b5625ade70b7f124623182ee2d784a4c1eaf3773ebd"}
Oct 30 03:41:30.040183 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.040132    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" podStartSLOduration=2.040116191 podStartE2EDuration="2.040116191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:30.039088786 +0000 UTC m=+3648.263338064" watchObservedRunningTime="2026-10-30 03:41:30.040116191 +0000 UTC m=+3648.264365488"
Oct 30 03:41:30.076331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.076284    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" podStartSLOduration=2.076269447 podStartE2EDuration="2.076269447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:30.076113782 +0000 UTC m=+3648.300363058" watchObservedRunningTime="2026-10-30 03:41:30.076269447 +0000 UTC m=+3648.300518724"
Oct 30 03:41:30.606725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.606689    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:30.610319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.610295    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.631613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.631585    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:30.648671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.648634    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:30.661007 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.660904    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:41:30.674882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.674853    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:41:30.675022 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.674913    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:41:30.675022 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.674932    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:41:30.675128 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.675030    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:41:30.716649 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.716621    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.716783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.716673    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqss\" (UniqueName: \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.716880 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.716800    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.716945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.716887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.716989 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.716940    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.773693 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.773653    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:30.817869 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.817839    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqss\" (UniqueName: \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.818053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.817887    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.818053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.817909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.818053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.817937    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.818053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.817955    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.818636 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.818605    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.820494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.820471    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.820574 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.820475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.820609 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.820595    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.841716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.841687    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqss\" (UniqueName: \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\") pod \"data-science-collector-collector-1\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:30.885761 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.885733    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:30.927517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:30.927489    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:41:31.223060 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:31.222979    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:41:31.227733 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:41:31.227707    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568c598c_b5a8_480a_ae92_55064dbcd0ba.slice/crio-e9bd9e6c7a8e35ad0a9f68b047da6c30f556c59908e628c7e0c84f2f33ea92fb WatchSource:0}: Error finding container e9bd9e6c7a8e35ad0a9f68b047da6c30f556c59908e628c7e0c84f2f33ea92fb: Status 404 returned error can't find the container with id e9bd9e6c7a8e35ad0a9f68b047da6c30f556c59908e628c7e0c84f2f33ea92fb
Oct 30 03:41:31.230492 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:31.230462    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:31.230591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:31.230532    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:41:32.011902 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:32.011860    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"568c598c-b5a8-480a-ae92-55064dbcd0ba","Type":"ContainerStarted","Data":"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"}
Oct 30 03:41:32.012318 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:32.011920    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"568c598c-b5a8-480a-ae92-55064dbcd0ba","Type":"ContainerStarted","Data":"e9bd9e6c7a8e35ad0a9f68b047da6c30f556c59908e628c7e0c84f2f33ea92fb"}
Oct 30 03:41:32.050949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:32.050897    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.050883216 podStartE2EDuration="2.050883216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:32.050408905 +0000 UTC m=+3650.274658182" watchObservedRunningTime="2026-10-30 03:41:32.050883216 +0000 UTC m=+3650.275132492"
Oct 30 03:41:33.510136 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:33.510094    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:33.759303 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:33.759268    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:33.875169 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:33.875078    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:34.126689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.126604    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:34.245296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.245258    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:34.247209 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.247188    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:41:34.258790 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.258755    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.268928 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.268902    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:41:34.277579 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.277547    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 03:41:34.277734 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.277598    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 03:41:34.277926 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.277903    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 03:41:34.339549 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339516    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 03:41:34.339732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339623    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 03:41:34.339875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339723    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 03:41:34.339875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339746    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 03:41:34.339875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339758    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 03:41:34.339875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339785    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 03:41:34.340019 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.339894    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-hmf28\""
Oct 30 03:41:34.350357 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350332    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350362    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350383    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350402    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350660 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350546    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350660 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350583    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350660 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350616    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350762 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350762 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350762 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350743    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350890 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjk4v\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.350890 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.350805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451707 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451707 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451707    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451864    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.451993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.451952    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjk4v\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452056    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452085    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452153    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452510 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452395    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.452964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.452934    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.453278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.453249    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.454725 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.454703    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.454843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.454734    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.455106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.455083    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.455216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.455184    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.455520 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.455503    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.455668 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.455646    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.456225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.456194    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.456382 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.456366    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.473211 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:34.473189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjk4v\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:34.632438 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:41:34.632398    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:41:37.092783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:37.092746    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:39.058708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.058672    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:39.397353 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.397322    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.422010 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.421988    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.497875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.497837    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.498032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.497968    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.510062 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.510027    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.610050 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.610020    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.633945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.633918    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:39.680528 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:39.680456    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:40.010517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:40.010425    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:41:40.014522 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:41:40.014494    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded237b85_bb8d_4c84_94a3_2a0ef65954f4.slice/crio-c37146ff79432b75038a410e0463869058dd5af90115f3fab201f068f93dba55 WatchSource:0}: Error finding container c37146ff79432b75038a410e0463869058dd5af90115f3fab201f068f93dba55: Status 404 returned error can't find the container with id c37146ff79432b75038a410e0463869058dd5af90115f3fab201f068f93dba55
Oct 30 03:41:40.045811 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:40.045782    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerStarted","Data":"c37146ff79432b75038a410e0463869058dd5af90115f3fab201f068f93dba55"}
Oct 30 03:41:41.050898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:41.050852    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerStarted","Data":"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"}
Oct 30 03:41:44.064488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:44.064400    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b" exitCode=0
Oct 30 03:41:44.064488 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:44.064446    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerDied","Data":"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"}
Oct 30 03:41:45.070863 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:45.070804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerStarted","Data":"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"}
Oct 30 03:41:45.071255 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:45.070920    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerStarted","Data":"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"}
Oct 30 03:41:45.071255 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:45.070946    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerStarted","Data":"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"}
Oct 30 03:41:45.139615 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:45.139555    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.853684755 podStartE2EDuration="11.139543178s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285858423s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:45.129356536 +0000 UTC m=+3663.353605812" watchObservedRunningTime="2026-10-30 03:41:45.139543178 +0000 UTC m=+3663.363792452"
Oct 30 03:41:49.681088 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:49.681040    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:54.681389 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:54.681343    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:54.686602 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:54.686579    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:55.128128 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:55.128037    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:41:58.287953 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:58.287911    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:58.489399 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:58.489356    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:41:58.690911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:41:58.690870    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:00.134791 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:00.134755    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:00.135258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:00.134963    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="568c598c-b5a8-480a-ae92-55064dbcd0ba" containerName="otc-container" containerID="cri-o://91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e" gracePeriod=30
Oct 30 03:42:00.315898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:00.315864    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:00.440583 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:00.440483    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:00.558403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:00.558348    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:01.384987 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.384961    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:01.501304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501224    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\" (UniqueName: \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\") pod \"568c598c-b5a8-480a-ae92-55064dbcd0ba\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") "
Oct 30 03:42:01.501304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501279    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\" (UniqueName: \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\") pod \"568c598c-b5a8-480a-ae92-55064dbcd0ba\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") "
Oct 30 03:42:01.501304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\") pod \"568c598c-b5a8-480a-ae92-55064dbcd0ba\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") "
Oct 30 03:42:01.501591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501335    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\") pod \"568c598c-b5a8-480a-ae92-55064dbcd0ba\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") "
Oct 30 03:42:01.501591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\") pod \"568c598c-b5a8-480a-ae92-55064dbcd0ba\" (UID: \"568c598c-b5a8-480a-ae92-55064dbcd0ba\") "
Oct 30 03:42:01.501745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.501712    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal" pod "568c598c-b5a8-480a-ae92-55064dbcd0ba" (UID: "568c598c-b5a8-480a-ae92-55064dbcd0ba"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:42:01.503461 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.503436    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs" pod "568c598c-b5a8-480a-ae92-55064dbcd0ba" (UID: "568c598c-b5a8-480a-ae92-55064dbcd0ba"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.503542 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.503511    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs" pod "568c598c-b5a8-480a-ae92-55064dbcd0ba" (UID: "568c598c-b5a8-480a-ae92-55064dbcd0ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.503805 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.503778    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert" pod "568c598c-b5a8-480a-ae92-55064dbcd0ba" (UID: "568c598c-b5a8-480a-ae92-55064dbcd0ba"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:01.503805 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.503792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss" pod "568c598c-b5a8-480a-ae92-55064dbcd0ba" (UID: "568c598c-b5a8-480a-ae92-55064dbcd0ba"). InnerVolumeSpecName "kube-api-access-7wqss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:01.602242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.602203    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/568c598c-b5a8-480a-ae92-55064dbcd0ba-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.602242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.602234    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.602242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.602245    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.602482 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.602256    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/568c598c-b5a8-480a-ae92-55064dbcd0ba-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:42:01.602482 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:01.602267    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wqss\" (UniqueName: \"kubernetes.io/projected/568c598c-b5a8-480a-ae92-55064dbcd0ba-kube-api-access-7wqss\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:42:02.152722 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.152630    4528 generic.go:356] "Generic (PLEG): container finished" podID="568c598c-b5a8-480a-ae92-55064dbcd0ba" containerID="91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e" exitCode=0
Oct 30 03:42:02.152722 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.152679    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.152947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.152712    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"568c598c-b5a8-480a-ae92-55064dbcd0ba","Type":"ContainerDied","Data":"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"}
Oct 30 03:42:02.152947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.152776    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"568c598c-b5a8-480a-ae92-55064dbcd0ba","Type":"ContainerDied","Data":"e9bd9e6c7a8e35ad0a9f68b047da6c30f556c59908e628c7e0c84f2f33ea92fb"}
Oct 30 03:42:02.152947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.152799    4528 scope.go:122] "RemoveContainer" containerID="91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"
Oct 30 03:42:02.161175 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.161155    4528 scope.go:122] "RemoveContainer" containerID="91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"
Oct 30 03:42:02.161453 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:42:02.161424    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e\": container with ID starting with 91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e not found: ID does not exist" containerID="91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"
Oct 30 03:42:02.161523 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.161462    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e"} err="failed to get container status \"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e\": rpc error: code = NotFound desc = could not find container \"91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e\": container with ID starting with 91028e4e2d921c406188a5efdd9d8d3fde4186429fadd85d456d70e605c2037e not found: ID does not exist"
Oct 30 03:42:02.197412 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.197375    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:02.207274 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.207239    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:02.244130 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.244099    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:02.244416 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.244402    4528 cpu_manager.go:414] "containerMap: removing container" podUID="568c598c-b5a8-480a-ae92-55064dbcd0ba" containerName="otc-container"
Oct 30 03:42:02.244416 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.244416    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568c598c-b5a8-480a-ae92-55064dbcd0ba" containerName="otc-container"
Oct 30 03:42:02.244574 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.244502    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="568c598c-b5a8-480a-ae92-55064dbcd0ba" containerName="otc-container"
Oct 30 03:42:02.249051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.249028    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.261943 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.261912    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:42:02.261943 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.261917    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:42:02.262124 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.261986    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:42:02.262191 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.262176    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:42:02.262266 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.262250    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:42:02.272767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.272738    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:02.409375 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.409284    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.409375 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.409328    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.409792 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.409440    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.409792 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.409513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglbl\" (UniqueName: \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.409792 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.409544    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.447447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.447415    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:42:02.510503 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.510459    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.510648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.510511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.510648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.510544    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.510648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.510586    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglbl\" (UniqueName: \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.510648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.510622    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.511345 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.511319    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.513073 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.513049    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.513257 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.513233    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.513376 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.513255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.532007 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.531979    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglbl\" (UniqueName: \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\") pod \"data-science-collector-collector-1\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.561953 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.561913    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:42:02.859647 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.859546    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:42:02.864018 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:42:02.863991    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38781f8b_d437_420f_bc49_b2d14bf8f84c.slice/crio-b92a6400c273fa566226f0df75e37a5615dd82bc757689cfde77084e025964c5 WatchSource:0}: Error finding container b92a6400c273fa566226f0df75e37a5615dd82bc757689cfde77084e025964c5: Status 404 returned error can't find the container with id b92a6400c273fa566226f0df75e37a5615dd82bc757689cfde77084e025964c5
Oct 30 03:42:02.866432 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.866395    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:42:02.866509 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:02.866460    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:42:03.158861 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:03.158803    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"38781f8b-d437-420f-bc49-b2d14bf8f84c","Type":"ContainerStarted","Data":"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"}
Oct 30 03:42:03.159074 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:03.158922    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"38781f8b-d437-420f-bc49-b2d14bf8f84c","Type":"ContainerStarted","Data":"b92a6400c273fa566226f0df75e37a5615dd82bc757689cfde77084e025964c5"}
Oct 30 03:42:03.198366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:03.198298    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.198277879 podStartE2EDuration="1.198277879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:03.196732249 +0000 UTC m=+3681.420981550" watchObservedRunningTime="2026-10-30 03:42:03.198277879 +0000 UTC m=+3681.422527159"
Oct 30 03:42:03.893424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:03.893380    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568c598c-b5a8-480a-ae92-55064dbcd0ba" path="/var/lib/kubelet/pods/568c598c-b5a8-480a-ae92-55064dbcd0ba/volumes"
Oct 30 03:42:04.630770 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:42:04.630732    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:42:34.630417 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:42:34.630378    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:42:37.235359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:42:37.235323    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:04.630761 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:04.630681    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:43:25.578309 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:25.578273    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:26.590179 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:26.590140    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:26.708809 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:26.708768    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:26.909652 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:26.909616    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:27.033029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:27.032996    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:28.763491 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:28.763456    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:28.764017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:28.763659    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="38781f8b-d437-420f-bc49-b2d14bf8f84c" containerName="otc-container" containerID="cri-o://21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245" gracePeriod=30
Oct 30 03:43:28.954875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:28.954811    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:29.107234 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:29.107139    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:30.860625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.860589    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:30.864653 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.864629    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:30.877935 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.877901    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:43:30.877935 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.877901    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:43:30.887390 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887361    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:43:30.887390 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887382    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:43:30.887564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887385    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:43:30.887564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887405    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:43:30.887564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887411    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:43:30.887564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.887484    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-826kq\""
Oct 30 03:43:30.894741 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.894702    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:30.925388 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:30.925353    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:31.042173 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.042139    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:31.052485 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052447    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052603 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052519    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052603 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052582    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052729 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052729 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052773    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5h5r\" (UniqueName: \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.052978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.052888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.099187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.099153    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.153659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153608    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.153850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.153850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153759    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.153850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153789    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.153850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153864    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5h5r\" (UniqueName: \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.153945    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154540 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.154510    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154661 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.154516    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.154911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.154890    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.155278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.155258    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.156763 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.156739    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.157213 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.157195    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.157261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.157241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.175021 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.174986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5h5r\" (UniqueName: \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:31.255253 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255217    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\") pod \"38781f8b-d437-420f-bc49-b2d14bf8f84c\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") "
Oct 30 03:43:31.255443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255286    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\") pod \"38781f8b-d437-420f-bc49-b2d14bf8f84c\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") "
Oct 30 03:43:31.255443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255313    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\") pod \"38781f8b-d437-420f-bc49-b2d14bf8f84c\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") "
Oct 30 03:43:31.255443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255355    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\" (UniqueName: \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\") pod \"38781f8b-d437-420f-bc49-b2d14bf8f84c\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") "
Oct 30 03:43:31.255443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255403    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\") pod \"38781f8b-d437-420f-bc49-b2d14bf8f84c\" (UID: \"38781f8b-d437-420f-bc49-b2d14bf8f84c\") "
Oct 30 03:43:31.255943 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.255909    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal" pod "38781f8b-d437-420f-bc49-b2d14bf8f84c" (UID: "38781f8b-d437-420f-bc49-b2d14bf8f84c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:31.257701 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.257677    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl" pod "38781f8b-d437-420f-bc49-b2d14bf8f84c" (UID: "38781f8b-d437-420f-bc49-b2d14bf8f84c"). InnerVolumeSpecName "kube-api-access-dglbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:31.257853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.257835    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs" pod "38781f8b-d437-420f-bc49-b2d14bf8f84c" (UID: "38781f8b-d437-420f-bc49-b2d14bf8f84c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:31.258016 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.257995    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs" pod "38781f8b-d437-420f-bc49-b2d14bf8f84c" (UID: "38781f8b-d437-420f-bc49-b2d14bf8f84c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:31.258121 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.258010    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert" pod "38781f8b-d437-420f-bc49-b2d14bf8f84c" (UID: "38781f8b-d437-420f-bc49-b2d14bf8f84c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:31.356944 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.356908    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglbl\" (UniqueName: \"kubernetes.io/projected/38781f8b-d437-420f-bc49-b2d14bf8f84c-kube-api-access-dglbl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.356944 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.356942    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/38781f8b-d437-420f-bc49-b2d14bf8f84c-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.357148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.356955    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.357148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.356965    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.357148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.356975    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38781f8b-d437-420f-bc49-b2d14bf8f84c-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.514444 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.514354    4528 generic.go:356] "Generic (PLEG): container finished" podID="38781f8b-d437-420f-bc49-b2d14bf8f84c" containerID="21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245" exitCode=0
Oct 30 03:43:31.514444 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.514405    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.514645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.514437    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"38781f8b-d437-420f-bc49-b2d14bf8f84c","Type":"ContainerDied","Data":"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"}
Oct 30 03:43:31.514645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.514489    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"38781f8b-d437-420f-bc49-b2d14bf8f84c","Type":"ContainerDied","Data":"b92a6400c273fa566226f0df75e37a5615dd82bc757689cfde77084e025964c5"}
Oct 30 03:43:31.514645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.514512    4528 scope.go:122] "RemoveContainer" containerID="21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"
Oct 30 03:43:31.523743 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.523725    4528 scope.go:122] "RemoveContainer" containerID="21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"
Oct 30 03:43:31.524070 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:31.524049    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245\": container with ID starting with 21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245 not found: ID does not exist" containerID="21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"
Oct 30 03:43:31.524131 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.524080    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245"} err="failed to get container status \"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245\": rpc error: code = NotFound desc = could not find container \"21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245\": container with ID starting with 21f4950126ffd402ba4717651ad5461c2288f5d60c03b30ded53af1d84dca245 not found: ID does not exist"
Oct 30 03:43:31.557601 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.557569    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:31.564296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.564268    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:31.604655 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.604626    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:31.605054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.605017    4528 cpu_manager.go:414] "containerMap: removing container" podUID="38781f8b-d437-420f-bc49-b2d14bf8f84c" containerName="otc-container"
Oct 30 03:43:31.605054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.605039    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38781f8b-d437-420f-bc49-b2d14bf8f84c" containerName="otc-container"
Oct 30 03:43:31.605251 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.605134    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="38781f8b-d437-420f-bc49-b2d14bf8f84c" containerName="otc-container"
Oct 30 03:43:31.609571 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.609554    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.622387 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.622339    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\""
Oct 30 03:43:31.622387 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.622347    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:43:31.622548 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.622392    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:43:31.622548 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.622435    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:43:31.622548 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.622480    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:43:31.629441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.629418    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:31.761229 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.761186    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlbf\" (UniqueName: \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.761407 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.761294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.761407 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.761384    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.761509 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.761469    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.761591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.761571    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.862749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.862663    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.862749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.862716    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.862749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.862736    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlbf\" (UniqueName: \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.863226 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.862768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.863226 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.862793    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.863510 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.863489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.865243 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.865219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.865336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.865264    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.865336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.865321    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.883945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.883925    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlbf\" (UniqueName: \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\") pod \"data-science-collector-collector-1\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:31.892552 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.892528    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38781f8b-d437-420f-bc49-b2d14bf8f84c" path="/var/lib/kubelet/pods/38781f8b-d437-420f-bc49-b2d14bf8f84c/volumes"
Oct 30 03:43:31.922160 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:31.922140    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:32.219564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.219531    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:32.224962 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:43:32.224932    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952466f6_f542_41cd_bbca_3fa716386469.slice/crio-1a0619e6453f4b7317d6561466cc5b44b148283bf9fcbad468c0c01572678d41 WatchSource:0}: Error finding container 1a0619e6453f4b7317d6561466cc5b44b148283bf9fcbad468c0c01572678d41: Status 404 returned error can't find the container with id 1a0619e6453f4b7317d6561466cc5b44b148283bf9fcbad468c0c01572678d41
Oct 30 03:43:32.227502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.227467    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:32.227571 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.227529    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:32.520366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.520270    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"952466f6-f542-41cd-bbca-3fa716386469","Type":"ContainerStarted","Data":"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"}
Oct 30 03:43:32.520366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.520331    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"952466f6-f542-41cd-bbca-3fa716386469","Type":"ContainerStarted","Data":"1a0619e6453f4b7317d6561466cc5b44b148283bf9fcbad468c0c01572678d41"}
Oct 30 03:43:32.557897 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:32.557848    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.5578127579999999 podStartE2EDuration="1.557812758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:32.556790036 +0000 UTC m=+3770.781039338" watchObservedRunningTime="2026-10-30 03:43:32.557812758 +0000 UTC m=+3770.782062035"
Oct 30 03:43:34.630233 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:34.630188    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:43:35.294118 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.294082    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.318355 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.318318    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.395151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.395119    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.395327 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.395265    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.408174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.408148    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.508874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.508843    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.533059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.533028    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:35.682052 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:35.682019    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:36.006181 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.006084    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:36.010438 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:43:36.010411    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda573bb3d_e718_4f7c_979e_ce56034c588a.slice/crio-5e3cc7a3ba2fe4dcd30e49cb01d7f7c7e2c5d25c2929af756fce90866c5a4f0f WatchSource:0}: Error finding container 5e3cc7a3ba2fe4dcd30e49cb01d7f7c7e2c5d25c2929af756fce90866c5a4f0f: Status 404 returned error can't find the container with id 5e3cc7a3ba2fe4dcd30e49cb01d7f7c7e2c5d25c2929af756fce90866c5a4f0f
Oct 30 03:43:36.182295 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.182096    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:36.390266 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.390232    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:36.513584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.513547    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:36.541881 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.541844    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerStarted","Data":"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"}
Oct 30 03:43:36.542063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.541905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerStarted","Data":"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"}
Oct 30 03:43:36.542063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.541924    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerStarted","Data":"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"}
Oct 30 03:43:36.542063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.541946    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerStarted","Data":"5e3cc7a3ba2fe4dcd30e49cb01d7f7c7e2c5d25c2929af756fce90866c5a4f0f"}
Oct 30 03:43:36.542063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.541970    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:36.544632 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.544610    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:36.591955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:36.591896    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=6.591839935 podStartE2EDuration="6.591839935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:36.590531503 +0000 UTC m=+3774.814780779" watchObservedRunningTime="2026-10-30 03:43:36.591839935 +0000 UTC m=+3774.816089206"
Oct 30 03:43:37.328643 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:37.328607    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:37.546898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:37.546861    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:37.547104 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:37.546926    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:37.554813 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:37.554783    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:38.046324 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.046295    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:38.046498 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.046473    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="952466f6-f542-41cd-bbca-3fa716386469" containerName="otc-container" containerID="cri-o://c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace" gracePeriod=30
Oct 30 03:43:38.205990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.205959    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:38.288498 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.288473    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.343213 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.343115    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:38.426409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426373    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\") pod \"952466f6-f542-41cd-bbca-3fa716386469\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") "
Oct 30 03:43:38.426409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426413    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\" (UniqueName: \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\") pod \"952466f6-f542-41cd-bbca-3fa716386469\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") "
Oct 30 03:43:38.426631 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426567    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\") pod \"952466f6-f542-41cd-bbca-3fa716386469\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") "
Oct 30 03:43:38.426665 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426627    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\") pod \"952466f6-f542-41cd-bbca-3fa716386469\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") "
Oct 30 03:43:38.426696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426672    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\" (UniqueName: \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\") pod \"952466f6-f542-41cd-bbca-3fa716386469\" (UID: \"952466f6-f542-41cd-bbca-3fa716386469\") "
Oct 30 03:43:38.426809 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426787    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal" pod "952466f6-f542-41cd-bbca-3fa716386469" (UID: "952466f6-f542-41cd-bbca-3fa716386469"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:38.427008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.426987    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/952466f6-f542-41cd-bbca-3fa716386469-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:38.428749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.428720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert" pod "952466f6-f542-41cd-bbca-3fa716386469" (UID: "952466f6-f542-41cd-bbca-3fa716386469"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:38.428928 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.428903    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs" pod "952466f6-f542-41cd-bbca-3fa716386469" (UID: "952466f6-f542-41cd-bbca-3fa716386469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:38.429436 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.429406    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs" pod "952466f6-f542-41cd-bbca-3fa716386469" (UID: "952466f6-f542-41cd-bbca-3fa716386469"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:38.429648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.429626    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf" pod "952466f6-f542-41cd-bbca-3fa716386469" (UID: "952466f6-f542-41cd-bbca-3fa716386469"). InnerVolumeSpecName "kube-api-access-6tlbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:38.457323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.457286    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:38.527641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.527598    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:38.527641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.527639    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:38.527856 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.527656    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlbf\" (UniqueName: \"kubernetes.io/projected/952466f6-f542-41cd-bbca-3fa716386469-kube-api-access-6tlbf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:38.527856 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.527669    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/952466f6-f542-41cd-bbca-3fa716386469-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:38.552553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.552518    4528 generic.go:356] "Generic (PLEG): container finished" podID="952466f6-f542-41cd-bbca-3fa716386469" containerID="c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace" exitCode=0
Oct 30 03:43:38.552680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.552578    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.552680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.552606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"952466f6-f542-41cd-bbca-3fa716386469","Type":"ContainerDied","Data":"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"}
Oct 30 03:43:38.552747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.552680    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"952466f6-f542-41cd-bbca-3fa716386469","Type":"ContainerDied","Data":"1a0619e6453f4b7317d6561466cc5b44b148283bf9fcbad468c0c01572678d41"}
Oct 30 03:43:38.552747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.552707    4528 scope.go:122] "RemoveContainer" containerID="c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"
Oct 30 03:43:38.561671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.561654    4528 scope.go:122] "RemoveContainer" containerID="c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"
Oct 30 03:43:38.561969 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:38.561945    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace\": container with ID starting with c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace not found: ID does not exist" containerID="c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"
Oct 30 03:43:38.562033 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.561977    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace"} err="failed to get container status \"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace\": rpc error: code = NotFound desc = could not find container \"c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace\": container with ID starting with c1ce7384998675827ced2a829c4317841b5ae578fe4ae4a4d3859a1b655d3ace not found: ID does not exist"
Oct 30 03:43:38.596658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.596575    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:38.606498 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.606467    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:38.642788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.642753    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:38.643097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.643072    4528 cpu_manager.go:414] "containerMap: removing container" podUID="952466f6-f542-41cd-bbca-3fa716386469" containerName="otc-container"
Oct 30 03:43:38.643097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.643088    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952466f6-f542-41cd-bbca-3fa716386469" containerName="otc-container"
Oct 30 03:43:38.643314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.643226    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="952466f6-f542-41cd-bbca-3fa716386469" containerName="otc-container"
Oct 30 03:43:38.647697 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.647674    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.660706 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.660686    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:43:38.660859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.660734    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:43:38.660859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.660767    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:43:38.661199 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.661185    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:43:38.661591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.661576    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:43:38.671910 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.671884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:38.730266 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.730222    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.730464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.730294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.730464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.730417    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.730464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.730453    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8ws\" (UniqueName: \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.730638 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.730592    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832021 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.831978    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.832065    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.832117    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.832142    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8ws\" (UniqueName: \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.832209    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.832956 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.832931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.834797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.834777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.834982 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.834963    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.835140 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.835115    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.856007 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.855944    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8ws\" (UniqueName: \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\") pod \"data-science-collector-collector-1\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:38.960422 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:38.960392    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:39.226722 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.226687    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:39.279409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.279367    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:39.284303 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:43:39.284260    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91db2dc_50f4_4896_a69b_4bab2337b1bf.slice/crio-5f45fe502cab0892ae9cdf79872f14521f2680d719d022978451fa20d3630767 WatchSource:0}: Error finding container 5f45fe502cab0892ae9cdf79872f14521f2680d719d022978451fa20d3630767: Status 404 returned error can't find the container with id 5f45fe502cab0892ae9cdf79872f14521f2680d719d022978451fa20d3630767
Oct 30 03:43:39.286706 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.286670    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:39.286792 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.286747    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:39.295936 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.295909    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:39.560778 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.560691    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e91db2dc-50f4-4896-a69b-4bab2337b1bf","Type":"ContainerStarted","Data":"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"}
Oct 30 03:43:39.560778 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.560754    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e91db2dc-50f4-4896-a69b-4bab2337b1bf","Type":"ContainerStarted","Data":"5f45fe502cab0892ae9cdf79872f14521f2680d719d022978451fa20d3630767"}
Oct 30 03:43:39.561408 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.561106    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-826kq\" not found"
Oct 30 03:43:39.602302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.602252    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6022375210000002 podStartE2EDuration="1.602237521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:39.601187632 +0000 UTC m=+3777.825436908" watchObservedRunningTime="2026-10-30 03:43:39.602237521 +0000 UTC m=+3777.826486798"
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640390    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640446    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640473    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.14045 +0000 UTC m=+3778.364699276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640509    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140492264 +0000 UTC m=+3778.364741521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640406    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640551    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140539929 +0000 UTC m=+3778.364789185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640409    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640585    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140577654 +0000 UTC m=+3778.364826939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640624    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640636    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640674    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140663661 +0000 UTC m=+3778.364912916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:39.640703 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640691    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140685284 +0000 UTC m=+3778.364934538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:39.641314 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640802    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:39.641314 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:39.640889    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:40.140873481 +0000 UTC m=+3778.365122771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:39.891912 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:39.891810    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952466f6-f542-41cd-bbca-3fa716386469" path="/var/lib/kubelet/pods/952466f6-f542-41cd-bbca-3fa716386469/volumes"
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143400    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143427    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143446    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143438    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143493    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143477647 +0000 UTC m=+3779.367726901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:40.143492 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143452    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143510    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143503331 +0000 UTC m=+3779.367752585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143438    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143518    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143526    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143517346 +0000 UTC m=+3779.367766606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143549    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143535217 +0000 UTC m=+3779.367784485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143561    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143555496 +0000 UTC m=+3779.367804751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143571    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143565988 +0000 UTC m=+3779.367815243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:40.143806 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:40.143580    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:41.143575954 +0000 UTC m=+3779.367825210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:40.565203 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:40.565105    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo" containerID="cri-o://9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415" gracePeriod=30
Oct 30 03:43:40.565203 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:40.565125    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway" containerID="cri-o://df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448" gracePeriod=30
Oct 30 03:43:40.565676 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:40.565125    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway-opa" containerID="cri-o://64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729" gracePeriod=30
Oct 30 03:43:41.151371 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151335    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151393    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151417    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151396678 +0000 UTC m=+3781.375645947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151417    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151458    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151447656 +0000 UTC m=+3781.375696934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151457    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151466    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151335    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151488    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151488    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151471429 +0000 UTC m=+3781.375720695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151539    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151529135 +0000 UTC m=+3781.375778390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151551    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151544975 +0000 UTC m=+3781.375794230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:43:41.151566 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151574    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151555495 +0000 UTC m=+3781.375804750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:43:41.152037 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:41.151589    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac podName:a573bb3d-e718-4f7c-979e-ce56034c588a nodeName:}" failed. No retries permitted until 2026-10-30 03:43:43.151581282 +0000 UTC m=+3781.375830537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:43:41.543019 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.542916    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.47:8083/ready\": dial tcp 10.133.0.47:8083: connect: connection refused"
Oct 30 03:43:41.571409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.571369    4528 generic.go:356] "Generic (PLEG): container finished" podID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerID="64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729" exitCode=0
Oct 30 03:43:41.571409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.571399    4528 generic.go:356] "Generic (PLEG): container finished" podID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerID="9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415" exitCode=0
Oct 30 03:43:41.571802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.571443    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerDied","Data":"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"}
Oct 30 03:43:41.571802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.571496    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerDied","Data":"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"}
Oct 30 03:43:41.805112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.805054    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:41.856775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.856745    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.856975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.856848    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.856975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.856878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.856975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.856941    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.856974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\" (UniqueName: \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857167    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857198    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\") pod \"a573bb3d-e718-4f7c-979e-ce56034c588a\" (UID: \"a573bb3d-e718-4f7c-979e-ce56034c588a\") "
Oct 30 03:43:41.857415 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857328    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:41.857473 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857408    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:41.857546 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857526    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.857599 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857551    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.857991 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857933    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:41.858100 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.857988    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:41.859926 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.859869    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:41.860090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.859992    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:41.860090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.860070    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "kube-api-access-h5h5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:41.860300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.860274    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:41.860621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.860604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "a573bb3d-e718-4f7c-979e-ce56034c588a" (UID: "a573bb3d-e718-4f7c-979e-ce56034c588a"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:43:41.958906 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958873    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.958906 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958905    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.959156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958919    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5h5r\" (UniqueName: \"kubernetes.io/projected/a573bb3d-e718-4f7c-979e-ce56034c588a-kube-api-access-h5h5r\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.959156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958955    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:43:41.959156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958968    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.959156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958982    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.959156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.958997    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a573bb3d-e718-4f7c-979e-ce56034c588a-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:41.968562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:41.968546    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:43:42.059977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.059900    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:43:42.577520 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.577482    4528 generic.go:356] "Generic (PLEG): container finished" podID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerID="df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448" exitCode=0
Oct 30 03:43:42.577958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.577561    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:42.577958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.577569    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerDied","Data":"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"}
Oct 30 03:43:42.577958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.577657    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"a573bb3d-e718-4f7c-979e-ce56034c588a","Type":"ContainerDied","Data":"5e3cc7a3ba2fe4dcd30e49cb01d7f7c7e2c5d25c2929af756fce90866c5a4f0f"}
Oct 30 03:43:42.577958 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.577687    4528 scope.go:122] "RemoveContainer" containerID="64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"
Oct 30 03:43:42.585524 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.585502    4528 scope.go:122] "RemoveContainer" containerID="df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"
Oct 30 03:43:42.593044 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.593021    4528 scope.go:122] "RemoveContainer" containerID="9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"
Oct 30 03:43:42.600253 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.600230    4528 scope.go:122] "RemoveContainer" containerID="64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"
Oct 30 03:43:42.600513 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:42.600493    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729\": container with ID starting with 64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729 not found: ID does not exist" containerID="64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"
Oct 30 03:43:42.600556 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.600521    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729"} err="failed to get container status \"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729\": rpc error: code = NotFound desc = could not find container \"64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729\": container with ID starting with 64727213c7add8f0db80aad380bea3d607fbc21ee20fe678e49cf4ce2ffa7729 not found: ID does not exist"
Oct 30 03:43:42.600556 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.600538    4528 scope.go:122] "RemoveContainer" containerID="df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"
Oct 30 03:43:42.600760 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:42.600742    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448\": container with ID starting with df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448 not found: ID does not exist" containerID="df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"
Oct 30 03:43:42.600797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.600767    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448"} err="failed to get container status \"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448\": rpc error: code = NotFound desc = could not find container \"df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448\": container with ID starting with df93663fe00d1ab0a36b5ba8c7c3b7574a42f20ce9431d1f949dd4019ba8d448 not found: ID does not exist"
Oct 30 03:43:42.600797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.600782    4528 scope.go:122] "RemoveContainer" containerID="9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"
Oct 30 03:43:42.601055 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:42.601035    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415\": container with ID starting with 9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415 not found: ID does not exist" containerID="9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"
Oct 30 03:43:42.601119 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.601063    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415"} err="failed to get container status \"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415\": rpc error: code = NotFound desc = could not find container \"9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415\": container with ID starting with 9f391b57f8a95fdf2e17c5f7daceaaff8454296dd6bf663fbd071b2f36b3d415 not found: ID does not exist"
Oct 30 03:43:42.623044 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.623024    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:42.632968 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:42.632945    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:43.892084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:43.892053    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" path="/var/lib/kubelet/pods/a573bb3d-e718-4f7c-979e-ce56034c588a/volumes"
Oct 30 03:43:46.050475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:46.050437    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:46.201659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:46.201610    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:46.348330 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:46.348250    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:46.477187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:46.477150    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:48.158990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:48.158953    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:48.159494 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:48.159177    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" containerName="otc-container" containerID="cri-o://956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0" gracePeriod=30
Oct 30 03:43:48.336022 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:48.335981    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:48.467639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:48.467554    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:49.247101 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247068    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247483    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway-opa"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247501    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway-opa"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247529    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247537    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247549    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247556    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247661    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway-opa"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247672    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo"
Oct 30 03:43:49.247755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.247677    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a573bb3d-e718-4f7c-979e-ce56034c588a" containerName="tempo-gateway"
Oct 30 03:43:49.250808 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.250787    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.263635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.263614    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:43:49.263765 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.263643    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:43:49.263906 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.263702    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:43:49.273290 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.273273    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:43:49.278937 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.278868    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:49.311972 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.311846    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-rkdzf\""
Oct 30 03:43:49.311972 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.311896    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:43:49.311972 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.311843    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:43:49.311972 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.311933    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:43:49.315967 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.315856    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:49.405319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.405297    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.415929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.415898    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416074 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.415957    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416074 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416006    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416219 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416119    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416219 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfc7\" (UniqueName: \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416219 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416203    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416364 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416229    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.416364 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.416341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.501403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.501304    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:49.516646 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516621    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\") pod \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") "
Oct 30 03:43:49.516737 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516665    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\" (UniqueName: \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\") pod \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") "
Oct 30 03:43:49.516737 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516687    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\") pod \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") "
Oct 30 03:43:49.516886 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516751    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\") pod \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") "
Oct 30 03:43:49.516886 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516829    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\") pod \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\" (UID: \"e91db2dc-50f4-4896-a69b-4bab2337b1bf\") "
Oct 30 03:43:49.516988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516940    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.516988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.516970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517096 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517016    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517096 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517056    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfc7\" (UniqueName: \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517140    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517203    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.517298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517255    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.518079 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.518045    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.518278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.518254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.518350 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.517054    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal" pod "e91db2dc-50f4-4896-a69b-4bab2337b1bf" (UID: "e91db2dc-50f4-4896-a69b-4bab2337b1bf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:49.518749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.518720    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.518911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.518886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.519500 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.519472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs" pod "e91db2dc-50f4-4896-a69b-4bab2337b1bf" (UID: "e91db2dc-50f4-4896-a69b-4bab2337b1bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:49.519590 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.519507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs" pod "e91db2dc-50f4-4896-a69b-4bab2337b1bf" (UID: "e91db2dc-50f4-4896-a69b-4bab2337b1bf"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:49.519590 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.519546    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws" pod "e91db2dc-50f4-4896-a69b-4bab2337b1bf" (UID: "e91db2dc-50f4-4896-a69b-4bab2337b1bf"). InnerVolumeSpecName "kube-api-access-td8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:49.519939 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.519918    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.520181 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.520158    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.520661 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.520638    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert" pod "e91db2dc-50f4-4896-a69b-4bab2337b1bf" (UID: "e91db2dc-50f4-4896-a69b-4bab2337b1bf"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:49.520734 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.520719    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.538624 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.538595    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfc7\" (UniqueName: \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:49.609959 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.609926    4528 generic.go:356] "Generic (PLEG): container finished" podID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" containerID="956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0" exitCode=0
Oct 30 03:43:49.610122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.609973    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.610122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.609987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e91db2dc-50f4-4896-a69b-4bab2337b1bf","Type":"ContainerDied","Data":"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"}
Oct 30 03:43:49.610122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.610045    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e91db2dc-50f4-4896-a69b-4bab2337b1bf","Type":"ContainerDied","Data":"5f45fe502cab0892ae9cdf79872f14521f2680d719d022978451fa20d3630767"}
Oct 30 03:43:49.610122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.610067    4528 scope.go:122] "RemoveContainer" containerID="956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"
Oct 30 03:43:49.619493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.619462    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:49.619631 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.619500    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:49.619631 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.619517    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e91db2dc-50f4-4896-a69b-4bab2337b1bf-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:49.619631 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.619531    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8ws\" (UniqueName: \"kubernetes.io/projected/e91db2dc-50f4-4896-a69b-4bab2337b1bf-kube-api-access-td8ws\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:49.619631 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.619543    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91db2dc-50f4-4896-a69b-4bab2337b1bf-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:43:49.622256 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.622233    4528 scope.go:122] "RemoveContainer" containerID="956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"
Oct 30 03:43:49.622575 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:43:49.622555    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0\": container with ID starting with 956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0 not found: ID does not exist" containerID="956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"
Oct 30 03:43:49.622665 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.622586    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0"} err="failed to get container status \"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0\": rpc error: code = NotFound desc = could not find container \"956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0\": container with ID starting with 956f1a5be9a8cce148b7e9e261153723b05192cdff7b3b429b7ea7534b67d0f0 not found: ID does not exist"
Oct 30 03:43:49.659393 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.659354    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:49.668702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.668670    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:49.715959 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.715925    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:49.716332 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.716312    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" containerName="otc-container"
Oct 30 03:43:49.716417 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.716335    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" containerName="otc-container"
Oct 30 03:43:49.716475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.716445    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" containerName="otc-container"
Oct 30 03:43:49.721127 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.721107    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.734555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.734518    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:43:49.734751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.734569    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:43:49.734751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.734642    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:43:49.734751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.734704    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:43:49.734911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.734765    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-72c79cac\""
Oct 30 03:43:49.740744 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.740709    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:49.821720 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.821636    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.821720 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.821684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfdt\" (UniqueName: \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.821936 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.821787    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.821936 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.821886    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.821936 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.821921    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.891747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.891709    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91db2dc-50f4-4896-a69b-4bab2337b1bf" path="/var/lib/kubelet/pods/e91db2dc-50f4-4896-a69b-4bab2337b1bf/volumes"
Oct 30 03:43:49.923153 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923123    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfdt\" (UniqueName: \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.923287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923175    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.923287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923216    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.923287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.923422 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923320    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.924009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.923988    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.925752 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.925732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.925836 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.925764    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.925836 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.925809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:49.944201 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:49.944171    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfdt\" (UniqueName: \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\") pod \"data-science-collector-collector-1\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:50.033776 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.033745    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:43:50.334204 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.334122    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:50.339347 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:43:50.339320    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690ef890_ecd4_46b1_9d82_587604c98b1f.slice/crio-609843c3bf7d0cc93011e84a920a347be339586ee35bc644ba814ef810e56ad0 WatchSource:0}: Error finding container 609843c3bf7d0cc93011e84a920a347be339586ee35bc644ba814ef810e56ad0: Status 404 returned error can't find the container with id 609843c3bf7d0cc93011e84a920a347be339586ee35bc644ba814ef810e56ad0
Oct 30 03:43:50.341552 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.341525    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:50.341630 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.341584    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:43:50.617611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.617519    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"690ef890-ecd4-46b1-9d82-587604c98b1f","Type":"ContainerStarted","Data":"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"}
Oct 30 03:43:50.617611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.617579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"690ef890-ecd4-46b1-9d82-587604c98b1f","Type":"ContainerStarted","Data":"609843c3bf7d0cc93011e84a920a347be339586ee35bc644ba814ef810e56ad0"}
Oct 30 03:43:50.663770 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:50.663714    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.663701425 podStartE2EDuration="1.663701425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:50.663274183 +0000 UTC m=+3788.887523459" watchObservedRunningTime="2026-10-30 03:43:50.663701425 +0000 UTC m=+3788.887950744"
Oct 30 03:43:52.847256 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:52.847218    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:52.873949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:52.873918    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:52.948454 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:52.948428    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:52.948614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:52.948589    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:52.960915 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:52.960890    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:53.065787 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.065749    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:53.091372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.091343    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:53.175757 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.175719    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:53.508619 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.508529    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:53.513185 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:43:53.513151    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a4af82_613a_4aaa_a116_317e4c3f1790.slice/crio-955c99eef566c3909daef5e9b81d8fd0a1a513e06b795cc27f98372d0c5ba775 WatchSource:0}: Error finding container 955c99eef566c3909daef5e9b81d8fd0a1a513e06b795cc27f98372d0c5ba775: Status 404 returned error can't find the container with id 955c99eef566c3909daef5e9b81d8fd0a1a513e06b795cc27f98372d0c5ba775
Oct 30 03:43:53.635004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.634939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerStarted","Data":"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"}
Oct 30 03:43:53.635090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:53.635002    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerStarted","Data":"955c99eef566c3909daef5e9b81d8fd0a1a513e06b795cc27f98372d0c5ba775"}
Oct 30 03:43:54.641307 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:54.641272    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerStarted","Data":"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"}
Oct 30 03:43:54.641742 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:54.641323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerStarted","Data":"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"}
Oct 30 03:43:54.641742 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:54.641548    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:54.644238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:54.644218    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:54.706348 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:54.706278    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=5.7062585949999995 podStartE2EDuration="5.706258595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:54.692573405 +0000 UTC m=+3792.916822695" watchObservedRunningTime="2026-10-30 03:43:54.706258595 +0000 UTC m=+3792.930507873"
Oct 30 03:43:55.646663 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:55.646634    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:55.647144 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:55.646710    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:55.654147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:55.654124    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:43:55.836084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:55.835998    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:55.984629 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:55.984588    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:56.185193 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:56.185160    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:57.681424 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:57.681388    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:43:57.682134 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:57.681602    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="690ef890-ecd4-46b1-9d82-587604c98b1f" containerName="otc-container" containerID="cri-o://1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610" gracePeriod=30
Oct 30 03:43:57.854746 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:57.854701    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:57.985439 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:57.985356    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:58.843058 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:58.843020    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:43:58.843470 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:58.843279    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo" containerID="cri-o://39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e" gracePeriod=30
Oct 30 03:43:58.843655 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:58.843629    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway-opa" containerID="cri-o://f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c" gracePeriod=30
Oct 30 03:43:58.843723 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:58.843700    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway" containerID="cri-o://a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6" gracePeriod=30
Oct 30 03:43:58.934721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:58.934690    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:43:59.642553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:59.642502    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.50:8083/ready\": dial tcp 10.133.0.50:8083: connect: connection refused"
Oct 30 03:43:59.666761 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:59.666729    4528 generic.go:356] "Generic (PLEG): container finished" podID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerID="f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c" exitCode=0
Oct 30 03:43:59.666761 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:59.666753    4528 generic.go:356] "Generic (PLEG): container finished" podID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerID="39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e" exitCode=0
Oct 30 03:43:59.666977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:59.666804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerDied","Data":"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"}
Oct 30 03:43:59.666977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:43:59.666891    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerDied","Data":"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"}
Oct 30 03:44:00.036596 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.036565    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.087179 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.087148    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:00.215934 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.215840    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\" (UniqueName: \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\") pod \"690ef890-ecd4-46b1-9d82-587604c98b1f\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") "
Oct 30 03:44:00.215934 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.215886    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\") pod \"690ef890-ecd4-46b1-9d82-587604c98b1f\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.215992    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216032    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216053    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\") pod \"690ef890-ecd4-46b1-9d82-587604c98b1f\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216072    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\") pod \"690ef890-ecd4-46b1-9d82-587604c98b1f\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\" (UniqueName: \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216152 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216146    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\") pod \"690ef890-ecd4-46b1-9d82-587604c98b1f\" (UID: \"690ef890-ecd4-46b1-9d82-587604c98b1f\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216180    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216213    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216243    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216273    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216288    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal" pod "690ef890-ecd4-46b1-9d82-587604c98b1f" (UID: "690ef890-ecd4-46b1-9d82-587604c98b1f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216313    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.216480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216344    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\") pod \"74a4af82-613a-4aaa-a116-317e4c3f1790\" (UID: \"74a4af82-613a-4aaa-a116-317e4c3f1790\") "
Oct 30 03:44:00.217302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216924    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:00.217302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.216939    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/690ef890-ecd4-46b1-9d82-587604c98b1f-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.217302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.217176    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:00.217539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.217409    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:00.219117 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.219117 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219031    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert" pod "690ef890-ecd4-46b1-9d82-587604c98b1f" (UID: "690ef890-ecd4-46b1-9d82-587604c98b1f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.219117 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219098    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.219368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219121    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:00.219368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219224    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt" pod "690ef890-ecd4-46b1-9d82-587604c98b1f" (UID: "690ef890-ecd4-46b1-9d82-587604c98b1f"). InnerVolumeSpecName "kube-api-access-vzfdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:00.219555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.219555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:44:00.219954 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.219939    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7" pod "74a4af82-613a-4aaa-a116-317e4c3f1790" (UID: "74a4af82-613a-4aaa-a116-317e4c3f1790"). InnerVolumeSpecName "kube-api-access-8dfc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:00.220322 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.220309    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs" pod "690ef890-ecd4-46b1-9d82-587604c98b1f" (UID: "690ef890-ecd4-46b1-9d82-587604c98b1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.220403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.220389    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs" pod "690ef890-ecd4-46b1-9d82-587604c98b1f" (UID: "690ef890-ecd4-46b1-9d82-587604c98b1f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:00.317859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317778    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfdt\" (UniqueName: \"kubernetes.io/projected/690ef890-ecd4-46b1-9d82-587604c98b1f-kube-api-access-vzfdt\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.317859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317862    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317881    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317897    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317911    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317922    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfc7\" (UniqueName: \"kubernetes.io/projected/74a4af82-613a-4aaa-a116-317e4c3f1790-kube-api-access-8dfc7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317935    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/690ef890-ecd4-46b1-9d82-587604c98b1f-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317950    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317963    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317974    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.317985    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.318000    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.318107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.318013    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/74a4af82-613a-4aaa-a116-317e4c3f1790-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:00.327042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.327021    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:44:00.418735 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.418694    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:44:00.672051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.672018    4528 generic.go:356] "Generic (PLEG): container finished" podID="690ef890-ecd4-46b1-9d82-587604c98b1f" containerID="1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610" exitCode=0
Oct 30 03:44:00.672235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.672063    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.672235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.672093    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"690ef890-ecd4-46b1-9d82-587604c98b1f","Type":"ContainerDied","Data":"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"}
Oct 30 03:44:00.672235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.672139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"690ef890-ecd4-46b1-9d82-587604c98b1f","Type":"ContainerDied","Data":"609843c3bf7d0cc93011e84a920a347be339586ee35bc644ba814ef810e56ad0"}
Oct 30 03:44:00.672235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.672166    4528 scope.go:122] "RemoveContainer" containerID="1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"
Oct 30 03:44:00.674084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.674060    4528 generic.go:356] "Generic (PLEG): container finished" podID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerID="a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6" exitCode=0
Oct 30 03:44:00.674217 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.674103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerDied","Data":"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"}
Oct 30 03:44:00.674217 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.674134    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:00.674217 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.674136    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"74a4af82-613a-4aaa-a116-317e4c3f1790","Type":"ContainerDied","Data":"955c99eef566c3909daef5e9b81d8fd0a1a513e06b795cc27f98372d0c5ba775"}
Oct 30 03:44:00.681724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.681708    4528 scope.go:122] "RemoveContainer" containerID="1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"
Oct 30 03:44:00.681996 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:00.681975    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610\": container with ID starting with 1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610 not found: ID does not exist" containerID="1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"
Oct 30 03:44:00.682047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.682002    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610"} err="failed to get container status \"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610\": rpc error: code = NotFound desc = could not find container \"1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610\": container with ID starting with 1e74cb3877e9462a50bdd606a4ccceaf8fca8424f92e5d7081d11702158ee610 not found: ID does not exist"
Oct 30 03:44:00.682047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.682018    4528 scope.go:122] "RemoveContainer" containerID="f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"
Oct 30 03:44:00.691771 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.689970    4528 scope.go:122] "RemoveContainer" containerID="a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"
Oct 30 03:44:00.699253 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.699231    4528 scope.go:122] "RemoveContainer" containerID="39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"
Oct 30 03:44:00.706426 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.706408    4528 scope.go:122] "RemoveContainer" containerID="f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"
Oct 30 03:44:00.706693 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:00.706674    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c\": container with ID starting with f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c not found: ID does not exist" containerID="f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"
Oct 30 03:44:00.706738 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.706701    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c"} err="failed to get container status \"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c\": rpc error: code = NotFound desc = could not find container \"f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c\": container with ID starting with f1cfda3a4c9d013dda1b22a3d761b19a2130d4a4b83b30d53cba2e53fce0f76c not found: ID does not exist"
Oct 30 03:44:00.706738 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.706716    4528 scope.go:122] "RemoveContainer" containerID="a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"
Oct 30 03:44:00.706959 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:00.706942    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6\": container with ID starting with a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6 not found: ID does not exist" containerID="a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"
Oct 30 03:44:00.707009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.706963    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6"} err="failed to get container status \"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6\": rpc error: code = NotFound desc = could not find container \"a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6\": container with ID starting with a7d43c0737af46fe0885f8f7a96838faddc59bb62d8be712cde8fbf6a582f6a6 not found: ID does not exist"
Oct 30 03:44:00.707009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.706989    4528 scope.go:122] "RemoveContainer" containerID="39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"
Oct 30 03:44:00.707219 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:00.707202    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e\": container with ID starting with 39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e not found: ID does not exist" containerID="39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"
Oct 30 03:44:00.707269 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.707225    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e"} err="failed to get container status \"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e\": rpc error: code = NotFound desc = could not find container \"39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e\": container with ID starting with 39b909d80d0f014176099f4daa3670ed20460177b3890d19cc8678e3e1bb225e not found: ID does not exist"
Oct 30 03:44:00.717383 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.717358    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:00.727272 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.727245    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:00.753882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.753845    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:00.764226 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.764197    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:00.809478 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809448    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:00.809783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809754    4528 cpu_manager.go:414] "containerMap: removing container" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway"
Oct 30 03:44:00.809783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809769    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809796    4528 cpu_manager.go:414] "containerMap: removing container" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway-opa"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809802    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway-opa"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809812    4528 cpu_manager.go:414] "containerMap: removing container" podUID="690ef890-ecd4-46b1-9d82-587604c98b1f" containerName="otc-container"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809833    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690ef890-ecd4-46b1-9d82-587604c98b1f" containerName="otc-container"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809845    4528 cpu_manager.go:414] "containerMap: removing container" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809852    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809901    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809911    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809919    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" containerName="tempo-gateway-opa"
Oct 30 03:44:00.809966 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.809926    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="690ef890-ecd4-46b1-9d82-587604c98b1f" containerName="otc-container"
Oct 30 03:44:00.814390 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.814371    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.836062 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.836041    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:44:00.836671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.836647    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:00.836768 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.836660    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:00.836768 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.836725    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:00.836887 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.836833    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:00.842165 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.842145    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:00.922964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.922896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.922964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.922928    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcnf\" (UniqueName: \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.923123 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.922963    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.923123 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.923101    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:00.923215 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:00.923142    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.023745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.023701    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.023745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.023748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcnf\" (UniqueName: \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.023994 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.023793    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.023994 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.023862    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.023994 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.023901    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.024546 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.024519    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.026291 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.026271    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.026403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.026273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.026486 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.026464    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.048118 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.048086    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcnf\" (UniqueName: \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\") pod \"data-science-collector-collector-1\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.126413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.126380    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:01.426244 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.426138    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:01.429747 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:01.429721    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff26a8e_1dd8_408f_97b8_3a6846a9e3ab.slice/crio-1838d7ca5dd4c182222a3a99f23d2b37c022f550016adca084bf6d77bf030810 WatchSource:0}: Error finding container 1838d7ca5dd4c182222a3a99f23d2b37c022f550016adca084bf6d77bf030810: Status 404 returned error can't find the container with id 1838d7ca5dd4c182222a3a99f23d2b37c022f550016adca084bf6d77bf030810
Oct 30 03:44:01.431766 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.431735    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:01.431870 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.431796    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:01.681770 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.681677    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab","Type":"ContainerStarted","Data":"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"}
Oct 30 03:44:01.681770 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.681737    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab","Type":"ContainerStarted","Data":"1838d7ca5dd4c182222a3a99f23d2b37c022f550016adca084bf6d77bf030810"}
Oct 30 03:44:01.720641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.720591    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.7205781949999999 podStartE2EDuration="1.720578195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:01.719958669 +0000 UTC m=+3799.944207970" watchObservedRunningTime="2026-10-30 03:44:01.720578195 +0000 UTC m=+3799.944827471"
Oct 30 03:44:01.891577 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.891543    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690ef890-ecd4-46b1-9d82-587604c98b1f" path="/var/lib/kubelet/pods/690ef890-ecd4-46b1-9d82-587604c98b1f/volumes"
Oct 30 03:44:01.891997 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:01.891983    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a4af82-613a-4aaa-a116-317e4c3f1790" path="/var/lib/kubelet/pods/74a4af82-613a-4aaa-a116-317e4c3f1790/volumes"
Oct 30 03:44:04.631886 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:04.631838    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:44:04.761672 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:04.761630    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:04.982684 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:04.982598    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:05.181064 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:05.181013    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:06.114737 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.114699    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:06.681371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.681342    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:06.681610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.681582    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" containerName="otc-container" containerID="cri-o://41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec" gracePeriod=30
Oct 30 03:44:06.850877 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.850735    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:06.924682 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.924652    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:06.975140 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975058    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\") pod \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") "
Oct 30 03:44:06.975140 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975094    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\") pod \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") "
Oct 30 03:44:06.975140 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975130    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\" (UniqueName: \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\") pod \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") "
Oct 30 03:44:06.975440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975169    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\") pod \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") "
Oct 30 03:44:06.975440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975201    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\") pod \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\" (UID: \"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab\") "
Oct 30 03:44:06.975596 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.975572    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal" pod "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" (UID: "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:06.977416 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.977380    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf" pod "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" (UID: "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab"). InnerVolumeSpecName "kube-api-access-lkcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:06.977755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.977732    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs" pod "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" (UID: "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:06.977920 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.977753    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs" pod "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" (UID: "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:06.977920 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.977895    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert" pod "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" (UID: "0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:06.994610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:06.994579    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:07.076238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.076200    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:07.076238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.076229    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:07.076238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.076242    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcnf\" (UniqueName: \"kubernetes.io/projected/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-kube-api-access-lkcnf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:07.076461 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.076252    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:07.076461 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.076261    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:07.708340 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.708306    4528 generic.go:356] "Generic (PLEG): container finished" podID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" containerID="41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec" exitCode=0
Oct 30 03:44:07.708797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.708357    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.708797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.708366    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab","Type":"ContainerDied","Data":"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"}
Oct 30 03:44:07.708797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.708419    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab","Type":"ContainerDied","Data":"1838d7ca5dd4c182222a3a99f23d2b37c022f550016adca084bf6d77bf030810"}
Oct 30 03:44:07.708797 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.708451    4528 scope.go:122] "RemoveContainer" containerID="41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"
Oct 30 03:44:07.717393 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.717375    4528 scope.go:122] "RemoveContainer" containerID="41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"
Oct 30 03:44:07.717652 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:07.717630    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec\": container with ID starting with 41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec not found: ID does not exist" containerID="41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"
Oct 30 03:44:07.717724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.717660    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec"} err="failed to get container status \"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec\": rpc error: code = NotFound desc = could not find container \"41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec\": container with ID starting with 41b24d72078d279be5b4e01a2f9776c67a927b17b01ae0a62d2b9dee0075f0ec not found: ID does not exist"
Oct 30 03:44:07.752124 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.752087    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:07.761868 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.761843    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:07.801237 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.801208    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:07.801553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.801534    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" containerName="otc-container"
Oct 30 03:44:07.801553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.801549    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" containerName="otc-container"
Oct 30 03:44:07.801726 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.801636    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" containerName="otc-container"
Oct 30 03:44:07.806111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.806091    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.818729 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.818710    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-2dfe8e70\""
Oct 30 03:44:07.819092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.819073    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:07.819208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.819095    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:07.819270 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.819229    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:07.819307 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.819292    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:07.830287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.830261    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:07.882172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.882130    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bsp\" (UniqueName: \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.882172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.882177    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.882402 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.882320    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.882402 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.882360    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.882402 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.882382    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.892207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.892182    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab" path="/var/lib/kubelet/pods/0ff26a8e-1dd8-408f-97b8-3a6846a9e3ab/volumes"
Oct 30 03:44:07.983279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983193    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.983279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983238    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.983279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983262    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.983495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983284    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bsp\" (UniqueName: \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.983495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.983985 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.983954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.985804 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.985780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.985804 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.985790    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:07.985979 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:07.985874    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:08.004238 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.004215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bsp\" (UniqueName: \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\") pod \"data-science-collector-collector-1\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:08.118627 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.118586    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:08.417251 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.417211    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:08.421660 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:08.421626    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46c337b_5b1b_4795_b9f7_102e52696820.slice/crio-12c156e36fc06d2690e30f64473496011c3838d0dff4e56d050ae67a49fcd852 WatchSource:0}: Error finding container 12c156e36fc06d2690e30f64473496011c3838d0dff4e56d050ae67a49fcd852: Status 404 returned error can't find the container with id 12c156e36fc06d2690e30f64473496011c3838d0dff4e56d050ae67a49fcd852
Oct 30 03:44:08.423739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.423711    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:08.423813 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.423767    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:08.718033 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.717948    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c46c337b-5b1b-4795-b9f7-102e52696820","Type":"ContainerStarted","Data":"1e37d774d894050364d066cc5d477371332747a49bb095e8e240438566e674c0"}
Oct 30 03:44:08.718033 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.718008    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c46c337b-5b1b-4795-b9f7-102e52696820","Type":"ContainerStarted","Data":"12c156e36fc06d2690e30f64473496011c3838d0dff4e56d050ae67a49fcd852"}
Oct 30 03:44:08.755184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:08.755131    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.755117426 podStartE2EDuration="1.755117426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:08.754499338 +0000 UTC m=+3806.978748617" watchObservedRunningTime="2026-10-30 03:44:08.755117426 +0000 UTC m=+3806.979366703"
Oct 30 03:44:13.774838 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:13.774786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:13.979528 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:13.979484    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:14.134858 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:14.134809    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:15.560593 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.560560    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:15.561339 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.560752    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c46c337b-5b1b-4795-b9f7-102e52696820" containerName="otc-container" containerID="cri-o://1e37d774d894050364d066cc5d477371332747a49bb095e8e240438566e674c0" gracePeriod=30
Oct 30 03:44:15.752751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.752714    4528 generic.go:356] "Generic (PLEG): container finished" podID="c46c337b-5b1b-4795-b9f7-102e52696820" containerID="1e37d774d894050364d066cc5d477371332747a49bb095e8e240438566e674c0" exitCode=0
Oct 30 03:44:15.752941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.752746    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c46c337b-5b1b-4795-b9f7-102e52696820","Type":"ContainerDied","Data":"1e37d774d894050364d066cc5d477371332747a49bb095e8e240438566e674c0"}
Oct 30 03:44:15.776237 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.776202    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:15.801728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.801700    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:15.849505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.849308    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\" (UniqueName: \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\") pod \"c46c337b-5b1b-4795-b9f7-102e52696820\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") "
Oct 30 03:44:15.849505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.849377    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\") pod \"c46c337b-5b1b-4795-b9f7-102e52696820\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") "
Oct 30 03:44:15.849505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.849425    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\") pod \"c46c337b-5b1b-4795-b9f7-102e52696820\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") "
Oct 30 03:44:15.849505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.849458    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\") pod \"c46c337b-5b1b-4795-b9f7-102e52696820\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") "
Oct 30 03:44:15.849505 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.849491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\") pod \"c46c337b-5b1b-4795-b9f7-102e52696820\" (UID: \"c46c337b-5b1b-4795-b9f7-102e52696820\") "
Oct 30 03:44:15.850066 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.850036    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal" pod "c46c337b-5b1b-4795-b9f7-102e52696820" (UID: "c46c337b-5b1b-4795-b9f7-102e52696820"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:15.852765 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.852737    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp" pod "c46c337b-5b1b-4795-b9f7-102e52696820" (UID: "c46c337b-5b1b-4795-b9f7-102e52696820"). InnerVolumeSpecName "kube-api-access-87bsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:15.853287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.853052    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs" pod "c46c337b-5b1b-4795-b9f7-102e52696820" (UID: "c46c337b-5b1b-4795-b9f7-102e52696820"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:15.853287 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.853083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs" pod "c46c337b-5b1b-4795-b9f7-102e52696820" (UID: "c46c337b-5b1b-4795-b9f7-102e52696820"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:15.853438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.853418    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert" pod "c46c337b-5b1b-4795-b9f7-102e52696820" (UID: "c46c337b-5b1b-4795-b9f7-102e52696820"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:15.927215 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.927172    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:15.950386 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.950355    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bsp\" (UniqueName: \"kubernetes.io/projected/c46c337b-5b1b-4795-b9f7-102e52696820-kube-api-access-87bsp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:15.950386 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.950383    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:15.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.950397    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c46c337b-5b1b-4795-b9f7-102e52696820-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:15.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.950409    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:15.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:15.950423    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c46c337b-5b1b-4795-b9f7-102e52696820-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:16.758345 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.758308    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c46c337b-5b1b-4795-b9f7-102e52696820","Type":"ContainerDied","Data":"12c156e36fc06d2690e30f64473496011c3838d0dff4e56d050ae67a49fcd852"}
Oct 30 03:44:16.758743 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.758394    4528 scope.go:122] "RemoveContainer" containerID="1e37d774d894050364d066cc5d477371332747a49bb095e8e240438566e674c0"
Oct 30 03:44:16.758743 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.758322    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.806809 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.806777    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:16.836014 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.835984    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:16.914440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.914407    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:16.914803 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.914779    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c46c337b-5b1b-4795-b9f7-102e52696820" containerName="otc-container"
Oct 30 03:44:16.914803 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.914800    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46c337b-5b1b-4795-b9f7-102e52696820" containerName="otc-container"
Oct 30 03:44:16.915019 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.914880    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46c337b-5b1b-4795-b9f7-102e52696820" containerName="otc-container"
Oct 30 03:44:16.919532 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.919511    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.950955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.950931    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:16.951945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.951924    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:16.952157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.952138    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:16.957535 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.957511    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.957617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.957542    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.957675 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.957625    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.957713 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.957684    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.957782 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.957760    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqs5v\" (UniqueName: \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:16.958738 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.958719    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:16.958996 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.958978    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:16.959047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:16.959009    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:44:17.059010 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.058930    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.059010 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.058982    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.059303 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.059014    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.059303 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.059033    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqs5v\" (UniqueName: \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.059303 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.059066    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.059899 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.059866    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.061399 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.061378    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.061562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.061540    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.061620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.061606    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.079533 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.079511    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqs5v\" (UniqueName: \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\") pod \"data-science-collector-collector-1\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.231874 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.231838    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:17.530122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.530087    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:17.534969 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:17.534942    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf72c06_7ea7_44cc_8e15_a3259cec0d0d.slice/crio-8682cfe978abaf27839736c2077d0e3df2d5a716d38c030d3a3f62944b01437c WatchSource:0}: Error finding container 8682cfe978abaf27839736c2077d0e3df2d5a716d38c030d3a3f62944b01437c: Status 404 returned error can't find the container with id 8682cfe978abaf27839736c2077d0e3df2d5a716d38c030d3a3f62944b01437c
Oct 30 03:44:17.537029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.536996    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:17.537087 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.537059    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:17.765236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.765197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d","Type":"ContainerStarted","Data":"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"}
Oct 30 03:44:17.765614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.765262    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d","Type":"ContainerStarted","Data":"8682cfe978abaf27839736c2077d0e3df2d5a716d38c030d3a3f62944b01437c"}
Oct 30 03:44:17.804951 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.804851    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.804809001 podStartE2EDuration="1.804809001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:17.804309849 +0000 UTC m=+3816.028559123" watchObservedRunningTime="2026-10-30 03:44:17.804809001 +0000 UTC m=+3816.029058278"
Oct 30 03:44:17.893572 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:17.893541    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46c337b-5b1b-4795-b9f7-102e52696820" path="/var/lib/kubelet/pods/c46c337b-5b1b-4795-b9f7-102e52696820/volumes"
Oct 30 03:44:22.967082 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:22.967040    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:23.117224 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:23.117187    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:23.322978 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:23.322890    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:23.474857 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:23.474808    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:25.053639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:25.053525    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:25.054172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:25.053833    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" containerName="otc-container" containerID="cri-o://6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6" gracePeriod=30
Oct 30 03:44:25.210258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:25.210222    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:25.358297 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:25.358210    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:25.512184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:25.512147    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:26.350924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.350890    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:26.355112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.355087    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.375709 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.375684    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:44:26.375709 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.375696    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384222    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384271    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-wqr4f\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384287    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384305    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384295    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:44:26.385843 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.384275    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:44:26.386207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.386018    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:26.406981 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.406953    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:26.534720 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.534684    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:26.536686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536664    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.536745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.536745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.536866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.536905 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536875    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.536905 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536894    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.537009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.536991    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.537041 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.537020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgfz\" (UniqueName: \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638058    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638108    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638126    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638148    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638303    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638348    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgfz\" (UniqueName: \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.638905 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638872    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.639009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638926    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.639009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.638926    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.639218 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.639183    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.640618 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.640597    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.641115 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.641094    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.641255 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.641235    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:26.662333 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:26.662310    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgfz\" (UniqueName: \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:27.389833 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.389789    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:27.546892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.546786    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\" (UniqueName: \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\") pod \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") "
Oct 30 03:44:27.546892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.546851    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\") pod \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") "
Oct 30 03:44:27.546892 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.546887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\") pod \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") "
Oct 30 03:44:27.547174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.546936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\") pod \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") "
Oct 30 03:44:27.547174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.547017    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\") pod \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\" (UID: \"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d\") "
Oct 30 03:44:27.547518 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.547488    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal" pod "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" (UID: "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:27.549211 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.549178    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs" pod "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" (UID: "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:27.549361 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.549226    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert" pod "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" (UID: "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:27.549453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.549381    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v" pod "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" (UID: "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d"). InnerVolumeSpecName "kube-api-access-cqs5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:27.549453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.549397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs" pod "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" (UID: "fbf72c06-7ea7-44cc-8e15-a3259cec0d0d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:27.647929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.647888    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.647929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.647929    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.648078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.647940    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqs5v\" (UniqueName: \"kubernetes.io/projected/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-kube-api-access-cqs5v\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.648078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.647950    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.648078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.647960    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:27.820002 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.819916    4528 generic.go:356] "Generic (PLEG): container finished" podID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" containerID="6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6" exitCode=0
Oct 30 03:44:27.820002 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.819970    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:27.820198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.820002    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d","Type":"ContainerDied","Data":"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"}
Oct 30 03:44:27.820198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.820058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fbf72c06-7ea7-44cc-8e15-a3259cec0d0d","Type":"ContainerDied","Data":"8682cfe978abaf27839736c2077d0e3df2d5a716d38c030d3a3f62944b01437c"}
Oct 30 03:44:27.820198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.820082    4528 scope.go:122] "RemoveContainer" containerID="6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"
Oct 30 03:44:27.828403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.828384    4528 scope.go:122] "RemoveContainer" containerID="6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"
Oct 30 03:44:27.828674 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:27.828656    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6\": container with ID starting with 6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6 not found: ID does not exist" containerID="6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"
Oct 30 03:44:27.828721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.828683    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6"} err="failed to get container status \"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6\": rpc error: code = NotFound desc = could not find container \"6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6\": container with ID starting with 6453d2f6f0fe91444a1808f283c760d60d03c9724f3f013fe017977bb3c258e6 not found: ID does not exist"
Oct 30 03:44:27.859133 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.859099    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:27.867858 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.867831    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:27.876175 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.876152    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:27.892119 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.892096    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" path="/var/lib/kubelet/pods/fbf72c06-7ea7-44cc-8e15-a3259cec0d0d/volumes"
Oct 30 03:44:27.918867 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.918842    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:27.919268 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.919244    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" containerName="otc-container"
Oct 30 03:44:27.919361 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.919269    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" containerName="otc-container"
Oct 30 03:44:27.919405 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.919374    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf72c06-7ea7-44cc-8e15-a3259cec0d0d" containerName="otc-container"
Oct 30 03:44:27.923697 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.923679    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:27.941087 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.941067    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-45fbfa60\""
Oct 30 03:44:27.941612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.941594    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:27.941699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.941631    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:27.941699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.941681    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:27.941785 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.941717    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:27.950435 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:27.950412    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:28.050374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.050339    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.050546 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.050390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.050546 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.050516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzw7\" (UniqueName: \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.050620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.050556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.050620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.050587    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.151996 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.151958    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.151996 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.151994    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.152240 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.152025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.152240 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.152062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.152240 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.152111    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzw7\" (UniqueName: \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.152833 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.152782    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.154595 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.154569    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.154711 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.154643    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.154781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.154708    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.173558 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.173524    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzw7\" (UniqueName: \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\") pod \"data-science-collector-collector-1\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.235574 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.235538    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:28.534580 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.534489    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:28.539261 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:28.539233    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cf6105_4f5b_49e6_89b6_ba24498c5814.slice/crio-406bfaaa4511e0e31751ae1a9a512bc137060b7087edbc127dba2064c6ace6cb WatchSource:0}: Error finding container 406bfaaa4511e0e31751ae1a9a512bc137060b7087edbc127dba2064c6ace6cb: Status 404 returned error can't find the container with id 406bfaaa4511e0e31751ae1a9a512bc137060b7087edbc127dba2064c6ace6cb
Oct 30 03:44:28.541309 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.541280    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:28.541389 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.541339    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:28.826875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.826792    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c8cf6105-4f5b-49e6-89b6-ba24498c5814","Type":"ContainerStarted","Data":"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"}
Oct 30 03:44:28.826875 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.826858    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c8cf6105-4f5b-49e6-89b6-ba24498c5814","Type":"ContainerStarted","Data":"406bfaaa4511e0e31751ae1a9a512bc137060b7087edbc127dba2064c6ace6cb"}
Oct 30 03:44:28.869018 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:28.868936    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.868922613 podStartE2EDuration="1.868922613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:28.868355496 +0000 UTC m=+3827.092604774" watchObservedRunningTime="2026-10-30 03:44:28.868922613 +0000 UTC m=+3827.093171935"
Oct 30 03:44:32.739216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:32.739173    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:32.878220 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:32.878180    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:33.099533 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.099438    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:33.271182 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.271145    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:33.697380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.697347    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.722643 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.722615    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.798337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.798299    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.798775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.798424    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.811116 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.811080    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.913132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.913090    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:33.938401 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:33.938373    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.172442 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.172400    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.501432 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.501340    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:34.508805 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:34.508776    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cd1113_8463_4aa5_81c8_4ffa281c4676.slice/crio-81b5ea781dd73e34f70e16ffc93088eec4de4b2f7ab589c1df2959fc508233d4 WatchSource:0}: Error finding container 81b5ea781dd73e34f70e16ffc93088eec4de4b2f7ab589c1df2959fc508233d4: Status 404 returned error can't find the container with id 81b5ea781dd73e34f70e16ffc93088eec4de4b2f7ab589c1df2959fc508233d4
Oct 30 03:44:34.639171 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:34.639135    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:44:34.765171 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.765079    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:34.765326 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.765266    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" containerName="otc-container" containerID="cri-o://ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14" gracePeriod=30
Oct 30 03:44:34.855680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerStarted","Data":"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"}
Oct 30 03:44:34.856161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855697    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerStarted","Data":"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"}
Oct 30 03:44:34.856161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855717    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerStarted","Data":"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"}
Oct 30 03:44:34.856161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855733    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerStarted","Data":"81b5ea781dd73e34f70e16ffc93088eec4de4b2f7ab589c1df2959fc508233d4"}
Oct 30 03:44:34.856161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855849    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.856161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.855901    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.858423 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.858402    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.864778 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.864759    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:34.899608 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.899552    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=8.899534411 podStartE2EDuration="8.899534411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:34.899261104 +0000 UTC m=+3833.123510384" watchObservedRunningTime="2026-10-30 03:44:34.899534411 +0000 UTC m=+3833.123783691"
Oct 30 03:44:34.921278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:34.921242    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:35.014583 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.014557    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:35.067919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.067795    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:35.108270 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108242    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\" (UniqueName: \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\") pod \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") "
Oct 30 03:44:35.108431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108295    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\") pod \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") "
Oct 30 03:44:35.108431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\") pod \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") "
Oct 30 03:44:35.108431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108378    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\") pod \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") "
Oct 30 03:44:35.108431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108407    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\") pod \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\" (UID: \"c8cf6105-4f5b-49e6-89b6-ba24498c5814\") "
Oct 30 03:44:35.108772 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.108745    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal" pod "c8cf6105-4f5b-49e6-89b6-ba24498c5814" (UID: "c8cf6105-4f5b-49e6-89b6-ba24498c5814"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:35.110429 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.110409    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs" pod "c8cf6105-4f5b-49e6-89b6-ba24498c5814" (UID: "c8cf6105-4f5b-49e6-89b6-ba24498c5814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:35.110565 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.110524    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert" pod "c8cf6105-4f5b-49e6-89b6-ba24498c5814" (UID: "c8cf6105-4f5b-49e6-89b6-ba24498c5814"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:35.110860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.110809    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7" pod "c8cf6105-4f5b-49e6-89b6-ba24498c5814" (UID: "c8cf6105-4f5b-49e6-89b6-ba24498c5814"). InnerVolumeSpecName "kube-api-access-2gzw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:35.110860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.110813    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs" pod "c8cf6105-4f5b-49e6-89b6-ba24498c5814" (UID: "c8cf6105-4f5b-49e6-89b6-ba24498c5814"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:35.209830 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.209783    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzw7\" (UniqueName: \"kubernetes.io/projected/c8cf6105-4f5b-49e6-89b6-ba24498c5814-kube-api-access-2gzw7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:35.209830 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.209813    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:35.210014 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.209843    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:35.210014 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.209852    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c8cf6105-4f5b-49e6-89b6-ba24498c5814-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:35.210014 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.209860    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cf6105-4f5b-49e6-89b6-ba24498c5814-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:35.221200 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.221130    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:35.861768 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.861682    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" containerID="ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14" exitCode=0
Oct 30 03:44:35.861768 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.861732    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:35.862316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.861769    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c8cf6105-4f5b-49e6-89b6-ba24498c5814","Type":"ContainerDied","Data":"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"}
Oct 30 03:44:35.862316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.861838    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"c8cf6105-4f5b-49e6-89b6-ba24498c5814","Type":"ContainerDied","Data":"406bfaaa4511e0e31751ae1a9a512bc137060b7087edbc127dba2064c6ace6cb"}
Oct 30 03:44:35.862316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.861869    4528 scope.go:122] "RemoveContainer" containerID="ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"
Oct 30 03:44:35.862470 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.862358    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:35.871146 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.871131    4528 scope.go:122] "RemoveContainer" containerID="ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"
Oct 30 03:44:35.871427 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:35.871408    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14\": container with ID starting with ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14 not found: ID does not exist" containerID="ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"
Oct 30 03:44:35.871475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.871436    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14"} err="failed to get container status \"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14\": rpc error: code = NotFound desc = could not find container \"ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14\": container with ID starting with ccda15a0855ed2f58a62155cd222beb37121d13f146b7dd368c58c29ae8bbc14 not found: ID does not exist"
Oct 30 03:44:35.913299 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.913259    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:35.918358 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.918332    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:35.965208 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.965175    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:35.965709 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.965647    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" containerName="otc-container"
Oct 30 03:44:35.965709 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.965666    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" containerName="otc-container"
Oct 30 03:44:35.965803 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.965759    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" containerName="otc-container"
Oct 30 03:44:35.970288 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.970269    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:35.990846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.987736    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:35.990846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.987858    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:35.990846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.987885    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:44:35.990846 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:35.987903    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:36.000853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.000813    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:36.009964 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.009939    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:36.017342 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.017282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.017507 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.017414    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8tpw\" (UniqueName: \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.017583 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.017560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.017683 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.017658    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.017751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.017702    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.021748 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.021671    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:36.064586 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.064546    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:36.118645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.118566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.118645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.118617    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.118645 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.118639    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.118955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.118686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.118955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.118731    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8tpw\" (UniqueName: \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.119504 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.119473    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.121163 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.121141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.121267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.121215    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.121267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.121220    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.140911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.140887    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8tpw\" (UniqueName: \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\") pod \"data-science-collector-collector-1\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.286650 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.286599    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:36.591129 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.588762    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:36.593612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.593571    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:36.593712 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.593650    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:36.868877 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.868778    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"451e0d53-7656-4928-be66-4b2604970341","Type":"ContainerStarted","Data":"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"}
Oct 30 03:44:36.868877 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.868845    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"451e0d53-7656-4928-be66-4b2604970341","Type":"ContainerStarted","Data":"f3e340e98906bf15996f782e8bfbd5ad02b21d224ba9cad3152d47918f3623c0"}
Oct 30 03:44:36.869337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.869046    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-wqr4f\" not found"
Oct 30 03:44:36.910461 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:36.910411    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.91039984 podStartE2EDuration="1.91039984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:36.910091695 +0000 UTC m=+3835.134340973" watchObservedRunningTime="2026-10-30 03:44:36.91039984 +0000 UTC m=+3835.134649117"
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031621    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031652    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031692    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031728    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531704135 +0000 UTC m=+3835.755953408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031628    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031730    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031658    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031787    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531774959 +0000 UTC m=+3835.756024215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031664    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031807    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531797692 +0000 UTC m=+3835.756046949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031854    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531844543 +0000 UTC m=+3835.756093803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031873    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531864261 +0000 UTC m=+3835.756113516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031891    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531881852 +0000 UTC m=+3835.756131110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:37.031946 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.031928    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:37.531908998 +0000 UTC m=+3835.756158268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:37.473228 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:37.473186    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:37.537192 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537155    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:37.537192 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537180    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:37.537192 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537195    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537210    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537195    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537236    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537220676 +0000 UTC m=+3836.761469931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537197    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537253    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537245675 +0000 UTC m=+3836.761494931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537155    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537268    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537259433 +0000 UTC m=+3836.761508688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537318    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537301243 +0000 UTC m=+3836.761550509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537331    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537324 +0000 UTC m=+3836.761573254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537341    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.537335987 +0000 UTC m=+3836.761585242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:37.537433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:37.537355    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:38.5373502 +0000 UTC m=+3836.761599454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:37.874848 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:37.874742    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo" containerID="cri-o://ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d" gracePeriod=30
Oct 30 03:44:37.874848 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:37.874769    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway" containerID="cri-o://107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb" gracePeriod=30
Oct 30 03:44:37.875308 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:37.874754    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway-opa" containerID="cri-o://db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086" gracePeriod=30
Oct 30 03:44:37.892717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:37.892690    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cf6105-4f5b-49e6-89b6-ba24498c5814" path="/var/lib/kubelet/pods/c8cf6105-4f5b-49e6-89b6-ba24498c5814/volumes"
Oct 30 03:44:38.546803 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546764    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:38.546803 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546797    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546800    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546764    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546765    4528 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546778    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546774    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546865    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546843685 +0000 UTC m=+3838.771092957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546929    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546915668 +0000 UTC m=+3838.771164927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-gateway" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546940    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546934655 +0000 UTC m=+3838.771183909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-mtls" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546959    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546953834 +0000 UTC m=+3838.771203089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546968    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546963762 +0000 UTC m=+3838.771213017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-config" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546976    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546972042 +0000 UTC m=+3838.771221296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found
Oct 30 03:44:38.547036 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:38.546986    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle podName:f4cd1113-8463-4aa5-81c8-4ffa281c4676 nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.546981035 +0000 UTC m=+3838.771230290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found
Oct 30 03:44:38.881800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:38.881770    4528 generic.go:356] "Generic (PLEG): container finished" podID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerID="db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086" exitCode=0
Oct 30 03:44:38.881800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:38.881797    4528 generic.go:356] "Generic (PLEG): container finished" podID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerID="ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d" exitCode=0
Oct 30 03:44:38.882201 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:38.881854    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerDied","Data":"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"}
Oct 30 03:44:38.882201 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:38.881904    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerDied","Data":"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"}
Oct 30 03:44:39.116762 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.116734    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:39.254615 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254529    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254615 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254599    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\" (UniqueName: \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254624    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254685    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254806    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254854    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.254888 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.255225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.254946    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\" (UID: \"f4cd1113-8463-4aa5-81c8-4ffa281c4676\") "
Oct 30 03:44:39.255225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.255100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.255411 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.255241    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.255411 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.255361    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.255642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.255451    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.255642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.255615    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:39.257132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.257081    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "kube-api-access-gbgfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:39.257413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.257300    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.257413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.257355    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.257413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.257374    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:39.257739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.257722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "f4cd1113-8463-4aa5-81c8-4ffa281c4676" (UID: "f4cd1113-8463-4aa5-81c8-4ffa281c4676"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:44:39.356553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356513    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356546    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356557    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356567    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356578    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgfz\" (UniqueName: \"kubernetes.io/projected/f4cd1113-8463-4aa5-81c8-4ffa281c4676-kube-api-access-gbgfz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356587    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/f4cd1113-8463-4aa5-81c8-4ffa281c4676-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.356788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356614    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:44:39.356788 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.356625    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4cd1113-8463-4aa5-81c8-4ffa281c4676-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:39.368131 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.368113    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:44:39.457049 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.457013    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:44:39.888399 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.888368    4528 generic.go:356] "Generic (PLEG): container finished" podID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerID="107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb" exitCode=0
Oct 30 03:44:39.888789 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.888474    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:44:39.893563 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.893536    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerDied","Data":"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"}
Oct 30 03:44:39.893657 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.893590    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"f4cd1113-8463-4aa5-81c8-4ffa281c4676","Type":"ContainerDied","Data":"81b5ea781dd73e34f70e16ffc93088eec4de4b2f7ab589c1df2959fc508233d4"}
Oct 30 03:44:39.893657 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.893626    4528 scope.go:122] "RemoveContainer" containerID="db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"
Oct 30 03:44:39.902791 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.902769    4528 scope.go:122] "RemoveContainer" containerID="107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"
Oct 30 03:44:39.910328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.910306    4528 scope.go:122] "RemoveContainer" containerID="ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"
Oct 30 03:44:39.917456 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.917429    4528 scope.go:122] "RemoveContainer" containerID="db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"
Oct 30 03:44:39.917704 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:39.917686    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086\": container with ID starting with db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086 not found: ID does not exist" containerID="db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"
Oct 30 03:44:39.917758 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.917710    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086"} err="failed to get container status \"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086\": rpc error: code = NotFound desc = could not find container \"db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086\": container with ID starting with db00b6c76eb372e2495d3a233d8960a6257fce7a2fbeb6e8d8984c7b4a799086 not found: ID does not exist"
Oct 30 03:44:39.917758 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.917727    4528 scope.go:122] "RemoveContainer" containerID="107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"
Oct 30 03:44:39.918012 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:39.917994    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb\": container with ID starting with 107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb not found: ID does not exist" containerID="107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"
Oct 30 03:44:39.918067 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.918017    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb"} err="failed to get container status \"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb\": rpc error: code = NotFound desc = could not find container \"107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb\": container with ID starting with 107eddce98b4491428d9ec9f5329cc4a1967b58f1503486543156ec33e75d9eb not found: ID does not exist"
Oct 30 03:44:39.918067 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.918032    4528 scope.go:122] "RemoveContainer" containerID="ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"
Oct 30 03:44:39.918224 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:39.918210    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d\": container with ID starting with ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d not found: ID does not exist" containerID="ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"
Oct 30 03:44:39.918263 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.918228    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d"} err="failed to get container status \"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d\": rpc error: code = NotFound desc = could not find container \"ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d\": container with ID starting with ead833797ea04cd84853f0708aabd8824b320108304aecbb79e30b14dff4875d not found: ID does not exist"
Oct 30 03:44:39.943782 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.943751    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:39.953445 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:39.953424    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:44:41.892193 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:41.892159    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" path="/var/lib/kubelet/pods/f4cd1113-8463-4aa5-81c8-4ffa281c4676/volumes"
Oct 30 03:44:41.985334 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:41.985301    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:42.186794 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:42.186698    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:42.301620 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:42.301579    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:43.930127 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:43.930089    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:43.930634 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:43.930278    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="451e0d53-7656-4928-be66-4b2604970341" containerName="otc-container" containerID="cri-o://429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517" gracePeriod=30
Oct 30 03:44:44.098161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.098131    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:44.182949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.182889    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:44.194495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194471    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\") pod \"451e0d53-7656-4928-be66-4b2604970341\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") "
Oct 30 03:44:44.194610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194513    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\" (UniqueName: \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\") pod \"451e0d53-7656-4928-be66-4b2604970341\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") "
Oct 30 03:44:44.194610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194544    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\") pod \"451e0d53-7656-4928-be66-4b2604970341\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") "
Oct 30 03:44:44.194610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194585    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\") pod \"451e0d53-7656-4928-be66-4b2604970341\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") "
Oct 30 03:44:44.194610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194606    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\" (UniqueName: \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\") pod \"451e0d53-7656-4928-be66-4b2604970341\" (UID: \"451e0d53-7656-4928-be66-4b2604970341\") "
Oct 30 03:44:44.194850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.194805    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal" pod "451e0d53-7656-4928-be66-4b2604970341" (UID: "451e0d53-7656-4928-be66-4b2604970341"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:44.196641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.196610    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert" pod "451e0d53-7656-4928-be66-4b2604970341" (UID: "451e0d53-7656-4928-be66-4b2604970341"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.196767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.196672    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs" pod "451e0d53-7656-4928-be66-4b2604970341" (UID: "451e0d53-7656-4928-be66-4b2604970341"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.196767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.196690    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs" pod "451e0d53-7656-4928-be66-4b2604970341" (UID: "451e0d53-7656-4928-be66-4b2604970341"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:44.196767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.196751    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw" pod "451e0d53-7656-4928-be66-4b2604970341" (UID: "451e0d53-7656-4928-be66-4b2604970341"). InnerVolumeSpecName "kube-api-access-h8tpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:44.240783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.240753    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:44.295882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.295847    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.295882 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.295876    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8tpw\" (UniqueName: \"kubernetes.io/projected/451e0d53-7656-4928-be66-4b2604970341-kube-api-access-h8tpw\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.296068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.295888    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.296068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.295900    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/451e0d53-7656-4928-be66-4b2604970341-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.296068 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.295910    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/451e0d53-7656-4928-be66-4b2604970341-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:44.911783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.911748    4528 generic.go:356] "Generic (PLEG): container finished" podID="451e0d53-7656-4928-be66-4b2604970341" containerID="429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517" exitCode=0
Oct 30 03:44:44.911967 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.911793    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:44.911967 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.911837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"451e0d53-7656-4928-be66-4b2604970341","Type":"ContainerDied","Data":"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"}
Oct 30 03:44:44.911967 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.911880    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"451e0d53-7656-4928-be66-4b2604970341","Type":"ContainerDied","Data":"f3e340e98906bf15996f782e8bfbd5ad02b21d224ba9cad3152d47918f3623c0"}
Oct 30 03:44:44.911967 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.911906    4528 scope.go:122] "RemoveContainer" containerID="429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"
Oct 30 03:44:44.920325 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.920310    4528 scope.go:122] "RemoveContainer" containerID="429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"
Oct 30 03:44:44.920538 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:44.920518    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517\": container with ID starting with 429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517 not found: ID does not exist" containerID="429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"
Oct 30 03:44:44.920593 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.920550    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517"} err="failed to get container status \"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517\": rpc error: code = NotFound desc = could not find container \"429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517\": container with ID starting with 429e969db846020b103c59764b770933015e27af8d06b9a222321647fdc93517 not found: ID does not exist"
Oct 30 03:44:44.955183 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.955159    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:44.965313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.965279    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:44.997443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997412    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:44.997909 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997883    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo"
Oct 30 03:44:44.997909 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997908    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997922    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway-opa"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997930    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway-opa"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997953    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997960    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997984    4528 cpu_manager.go:414] "containerMap: removing container" podUID="451e0d53-7656-4928-be66-4b2604970341" containerName="otc-container"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.997992    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451e0d53-7656-4928-be66-4b2604970341" containerName="otc-container"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.998094    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.998104    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway-opa"
Oct 30 03:44:44.998112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.998115    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cd1113-8463-4aa5-81c8-4ffa281c4676" containerName="tempo-gateway"
Oct 30 03:44:44.998455 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:44.998127    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="451e0d53-7656-4928-be66-4b2604970341" containerName="otc-container"
Oct 30 03:44:45.002435 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.002415    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.017518 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.017497    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:45.020380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.020362    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:45.020507 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.020492    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:44:45.020552 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.020543    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:45.024893 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.024874    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:45.101110 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.101076    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.101110 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.101111    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.101395 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.101152    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.101395 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.101192    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cqg\" (UniqueName: \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.101395 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.101286    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.117954 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.117932    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:45.202545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.202457    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.202545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.202495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.202784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.202560    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.202784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.202586    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cqg\" (UniqueName: \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.202784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.202623    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.203328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.203302    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.205101 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.205077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.205239 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.205219    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.205281 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.205272    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.223961 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.223936    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cqg\" (UniqueName: \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\") pod \"data-science-collector-collector-1\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.315231 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.315191    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:45.617551 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.612004    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:45.621548 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:45.621522    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad3aa01_7c6f_4d56_a0ae_93e1c41f7bf4.slice/crio-6c9a41d59287ca5238b2ab94c74de7d20a8b7fa132b2b7b25c77215c593477ea WatchSource:0}: Error finding container 6c9a41d59287ca5238b2ab94c74de7d20a8b7fa132b2b7b25c77215c593477ea: Status 404 returned error can't find the container with id 6c9a41d59287ca5238b2ab94c74de7d20a8b7fa132b2b7b25c77215c593477ea
Oct 30 03:44:45.623556 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.623526    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:45.623610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.623584    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:45.893064 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.893029    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451e0d53-7656-4928-be66-4b2604970341" path="/var/lib/kubelet/pods/451e0d53-7656-4928-be66-4b2604970341/volumes"
Oct 30 03:44:45.918321 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.918284    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4","Type":"ContainerStarted","Data":"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"}
Oct 30 03:44:45.918485 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.918338    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4","Type":"ContainerStarted","Data":"6c9a41d59287ca5238b2ab94c74de7d20a8b7fa132b2b7b25c77215c593477ea"}
Oct 30 03:44:45.957172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:45.957114    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.957096867 podStartE2EDuration="1.957096867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:45.956757552 +0000 UTC m=+3844.181006829" watchObservedRunningTime="2026-10-30 03:44:45.957096867 +0000 UTC m=+3844.181346148"
Oct 30 03:44:50.601229 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:50.601182    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:50.801480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:50.801442    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:50.926613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:50.926581    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:52.467871 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.467839    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:52.468440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.468030    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" containerName="otc-container" containerID="cri-o://931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab" gracePeriod=30
Oct 30 03:44:52.667389 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.667351    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:52.704565 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.704539    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:52.766497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.766423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\") pod \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") "
Oct 30 03:44:52.766497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.766462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\" (UniqueName: \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\") pod \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") "
Oct 30 03:44:52.766719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.766523    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\") pod \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") "
Oct 30 03:44:52.766719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.766550    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\") pod \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") "
Oct 30 03:44:52.766719 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.766569    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\") pod \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\" (UID: \"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4\") "
Oct 30 03:44:52.767066 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.767039    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal" pod "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" (UID: "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:44:52.768805 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.768773    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs" pod "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" (UID: "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:52.768933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.768811    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert" pod "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" (UID: "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:52.769001 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.768982    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs" pod "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" (UID: "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:44:52.770118 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.770100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg" pod "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" (UID: "cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4"). InnerVolumeSpecName "kube-api-access-x6cqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:44:52.801524 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.801487    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:52.867740 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.867710    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:52.867740 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.867737    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:52.867988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.867749    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:52.867988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.867760    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cqg\" (UniqueName: \"kubernetes.io/projected/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-kube-api-access-x6cqg\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:52.867988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.867772    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:44:52.947179 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.947135    4528 generic.go:356] "Generic (PLEG): container finished" podID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" containerID="931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab" exitCode=0
Oct 30 03:44:52.947368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.947195    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:52.947368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.947220    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4","Type":"ContainerDied","Data":"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"}
Oct 30 03:44:52.947368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.947285    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4","Type":"ContainerDied","Data":"6c9a41d59287ca5238b2ab94c74de7d20a8b7fa132b2b7b25c77215c593477ea"}
Oct 30 03:44:52.947368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.947315    4528 scope.go:122] "RemoveContainer" containerID="931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"
Oct 30 03:44:52.955753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.955732    4528 scope.go:122] "RemoveContainer" containerID="931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"
Oct 30 03:44:52.956020 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:44:52.956000    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab\": container with ID starting with 931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab not found: ID does not exist" containerID="931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"
Oct 30 03:44:52.956093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.956032    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab"} err="failed to get container status \"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab\": rpc error: code = NotFound desc = could not find container \"931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab\": container with ID starting with 931e9617e02b062bf44828ab68a4a48eee7b6269657f70c5040cb2b8e4c87fab not found: ID does not exist"
Oct 30 03:44:52.986802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.986760    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:52.990973 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.990950    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:52.996235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:52.996214    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:53.034342 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.034261    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:53.034607 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.034590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" containerName="otc-container"
Oct 30 03:44:53.034763 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.034606    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" containerName="otc-container"
Oct 30 03:44:53.034763 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.034679    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" containerName="otc-container"
Oct 30 03:44:53.039017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.038998    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.051463 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.051445    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:44:53.052344 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.052325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:44:53.052344 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.052335    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:44:53.052460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.052340    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:44:53.052460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.052340    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:44:53.060216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.059887    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:53.073921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.070333    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.073921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.070395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.073921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.070427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.073921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.070470    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.073921 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.070514    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4jr\" (UniqueName: \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.171148 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.171106    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.171318 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.171168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.171318 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.171204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.171417 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.171376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.171669 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.171434    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4jr\" (UniqueName: \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.172037 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.172012    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.173905 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.173880    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.174157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.174139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.174210 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.174155    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.192434 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.192415    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4jr\" (UniqueName: \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\") pod \"data-science-collector-collector-1\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.350997 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.350919    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:44:53.646058 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.646019    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:44:53.650556 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:44:53.650525    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b38bbc_a95e_425a_8e63_968694837db0.slice/crio-d0499b37ec11d3006fd13373678044e342988dc306a56008d1a3b352f90eeadc WatchSource:0}: Error finding container d0499b37ec11d3006fd13373678044e342988dc306a56008d1a3b352f90eeadc: Status 404 returned error can't find the container with id d0499b37ec11d3006fd13373678044e342988dc306a56008d1a3b352f90eeadc
Oct 30 03:44:53.652747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.652706    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:53.652885 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.652775    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:44:53.892460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.892428    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4" path="/var/lib/kubelet/pods/cad3aa01-7c6f-4d56-a0ae-93e1c41f7bf4/volumes"
Oct 30 03:44:53.952677 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.952596    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"57b38bbc-a95e-425a-8e63-968694837db0","Type":"ContainerStarted","Data":"77a655060b76d7af550ddd8b343e327de1cc9b54f60b0e7e2b29738ad3db3271"}
Oct 30 03:44:53.952677 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.952654    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"57b38bbc-a95e-425a-8e63-968694837db0","Type":"ContainerStarted","Data":"d0499b37ec11d3006fd13373678044e342988dc306a56008d1a3b352f90eeadc"}
Oct 30 03:44:53.990158 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:53.990108    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=0.990097097 podStartE2EDuration="990.097097ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:53.989219245 +0000 UTC m=+3852.213468522" watchObservedRunningTime="2026-10-30 03:44:53.990097097 +0000 UTC m=+3852.214346373"
Oct 30 03:44:55.269704 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:55.269671    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:55.391704 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:55.391674    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:59.161302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:59.161264    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:59.374582 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:59.374550    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:59.581269 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:59.581192    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:44:59.703313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:44:59.703275    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:01.259004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:01.258964    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:01.388428 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:01.388388    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:01.534091 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:01.534010    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:03.383931 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:03.383898    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:04.633315 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:45:04.633279    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:45:06.888522 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:06.887617    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:06.888522 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:06.887886    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="57b38bbc-a95e-425a-8e63-968694837db0" containerName="otc-container" containerID="cri-o://77a655060b76d7af550ddd8b343e327de1cc9b54f60b0e7e2b29738ad3db3271" gracePeriod=30
Oct 30 03:45:07.859706 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:07.859676    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:08.012087 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.012058    4528 generic.go:356] "Generic (PLEG): container finished" podID="57b38bbc-a95e-425a-8e63-968694837db0" containerID="77a655060b76d7af550ddd8b343e327de1cc9b54f60b0e7e2b29738ad3db3271" exitCode=0
Oct 30 03:45:08.012452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.012095    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"57b38bbc-a95e-425a-8e63-968694837db0","Type":"ContainerDied","Data":"77a655060b76d7af550ddd8b343e327de1cc9b54f60b0e7e2b29738ad3db3271"}
Oct 30 03:45:08.059387 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.059356    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:08.141348 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.141324    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:08.193699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.193666    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:08.200562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.200532    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\" (UniqueName: \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\") pod \"57b38bbc-a95e-425a-8e63-968694837db0\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") "
Oct 30 03:45:08.200710 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.200571    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\") pod \"57b38bbc-a95e-425a-8e63-968694837db0\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") "
Oct 30 03:45:08.200710 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.200599    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\") pod \"57b38bbc-a95e-425a-8e63-968694837db0\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") "
Oct 30 03:45:08.200710 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.200681    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\") pod \"57b38bbc-a95e-425a-8e63-968694837db0\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") "
Oct 30 03:45:08.200908 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.200728    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\") pod \"57b38bbc-a95e-425a-8e63-968694837db0\" (UID: \"57b38bbc-a95e-425a-8e63-968694837db0\") "
Oct 30 03:45:08.201369 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.201328    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal" pod "57b38bbc-a95e-425a-8e63-968694837db0" (UID: "57b38bbc-a95e-425a-8e63-968694837db0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:08.203241 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.203210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr" pod "57b38bbc-a95e-425a-8e63-968694837db0" (UID: "57b38bbc-a95e-425a-8e63-968694837db0"). InnerVolumeSpecName "kube-api-access-zp4jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:08.203241 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.203237    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs" pod "57b38bbc-a95e-425a-8e63-968694837db0" (UID: "57b38bbc-a95e-425a-8e63-968694837db0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:08.203383 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.203258    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs" pod "57b38bbc-a95e-425a-8e63-968694837db0" (UID: "57b38bbc-a95e-425a-8e63-968694837db0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:08.203383 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.203290    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert" pod "57b38bbc-a95e-425a-8e63-968694837db0" (UID: "57b38bbc-a95e-425a-8e63-968694837db0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:08.301807 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.301769    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/57b38bbc-a95e-425a-8e63-968694837db0-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:08.301807 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.301803    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:08.302027 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.301837    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4jr\" (UniqueName: \"kubernetes.io/projected/57b38bbc-a95e-425a-8e63-968694837db0-kube-api-access-zp4jr\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:08.302027 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.301871    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:08.302027 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:08.301885    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/57b38bbc-a95e-425a-8e63-968694837db0-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:09.022871 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.022836    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.023336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.022842    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"57b38bbc-a95e-425a-8e63-968694837db0","Type":"ContainerDied","Data":"d0499b37ec11d3006fd13373678044e342988dc306a56008d1a3b352f90eeadc"}
Oct 30 03:45:09.023336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.023000    4528 scope.go:122] "RemoveContainer" containerID="77a655060b76d7af550ddd8b343e327de1cc9b54f60b0e7e2b29738ad3db3271"
Oct 30 03:45:09.070728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.070688    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.080198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.080167    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.117403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.117377    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.117765 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.117742    4528 cpu_manager.go:414] "containerMap: removing container" podUID="57b38bbc-a95e-425a-8e63-968694837db0" containerName="otc-container"
Oct 30 03:45:09.117765 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.117760    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b38bbc-a95e-425a-8e63-968694837db0" containerName="otc-container"
Oct 30 03:45:09.117908 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.117851    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b38bbc-a95e-425a-8e63-968694837db0" containerName="otc-container"
Oct 30 03:45:09.122300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.122280    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.139246 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.139223    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:45:09.139246 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.139225    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:45:09.139396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.139245    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:45:09.139396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.139265    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:45:09.139396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.139332    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:45:09.149537 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.149509    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.210371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.210341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.210371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.210372    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.210614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.210393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4w8s\" (UniqueName: \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.210614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.210557    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.210721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.210660    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.311517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.311425    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.311517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.311479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.311517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.311503    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.311781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.311528    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4w8s\" (UniqueName: \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.311781 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.311592    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.312199 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.312173    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.314020 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.313992    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.314130 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.314092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.314130 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.314114    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.332519 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.332497    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4w8s\" (UniqueName: \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\") pod \"data-science-collector-collector-1\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.434443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.434407    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:09.728008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.727975    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.732805 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:45:09.732777    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c738099_89b4_4180_b2ef_7ea43062c57f.slice/crio-bf62fdfbaa438746296e9cc1e2fe9bc6d7c595b43bff683d937eb8bd69e19f3b WatchSource:0}: Error finding container bf62fdfbaa438746296e9cc1e2fe9bc6d7c595b43bff683d937eb8bd69e19f3b: Status 404 returned error can't find the container with id bf62fdfbaa438746296e9cc1e2fe9bc6d7c595b43bff683d937eb8bd69e19f3b
Oct 30 03:45:09.734868 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.734841    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:09.734934 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.734897    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:09.829100 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.829064    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:09.892233 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.892203    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b38bbc-a95e-425a-8e63-968694837db0" path="/var/lib/kubelet/pods/57b38bbc-a95e-425a-8e63-968694837db0/volumes"
Oct 30 03:45:09.984229 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:09.984139    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:10.029441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:10.029408    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c738099-89b4-4180-b2ef-7ea43062c57f","Type":"ContainerStarted","Data":"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"}
Oct 30 03:45:10.029955 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:10.029467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c738099-89b4-4180-b2ef-7ea43062c57f","Type":"ContainerStarted","Data":"bf62fdfbaa438746296e9cc1e2fe9bc6d7c595b43bff683d937eb8bd69e19f3b"}
Oct 30 03:45:10.079587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:10.079531    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.079516824 podStartE2EDuration="1.079516824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:10.07831924 +0000 UTC m=+3868.302568528" watchObservedRunningTime="2026-10-30 03:45:10.079516824 +0000 UTC m=+3868.303766146"
Oct 30 03:45:10.121696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:10.121658    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:10.237949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:10.237867    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:11.033046 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.033008    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4c738099-89b4-4180-b2ef-7ea43062c57f" containerName="otc-container" containerID="cri-o://2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764" gracePeriod=30
Oct 30 03:45:11.270656 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.270629    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:11.327408 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\") pod \"4c738099-89b4-4180-b2ef-7ea43062c57f\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") "
Oct 30 03:45:11.327408 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327381    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\") pod \"4c738099-89b4-4180-b2ef-7ea43062c57f\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") "
Oct 30 03:45:11.327609 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327426    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\") pod \"4c738099-89b4-4180-b2ef-7ea43062c57f\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") "
Oct 30 03:45:11.327609 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\" (UniqueName: \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\") pod \"4c738099-89b4-4180-b2ef-7ea43062c57f\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") "
Oct 30 03:45:11.327609 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327533    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\") pod \"4c738099-89b4-4180-b2ef-7ea43062c57f\" (UID: \"4c738099-89b4-4180-b2ef-7ea43062c57f\") "
Oct 30 03:45:11.327782 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.327739    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal" pod "4c738099-89b4-4180-b2ef-7ea43062c57f" (UID: "4c738099-89b4-4180-b2ef-7ea43062c57f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:11.330092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.330066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert" pod "4c738099-89b4-4180-b2ef-7ea43062c57f" (UID: "4c738099-89b4-4180-b2ef-7ea43062c57f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:11.330092 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.330077    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs" pod "4c738099-89b4-4180-b2ef-7ea43062c57f" (UID: "4c738099-89b4-4180-b2ef-7ea43062c57f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:11.330710 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.330643    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s" pod "4c738099-89b4-4180-b2ef-7ea43062c57f" (UID: "4c738099-89b4-4180-b2ef-7ea43062c57f"). InnerVolumeSpecName "kube-api-access-z4w8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:11.330981 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.330961    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs" pod "4c738099-89b4-4180-b2ef-7ea43062c57f" (UID: "4c738099-89b4-4180-b2ef-7ea43062c57f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:11.428610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.428574    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4c738099-89b4-4180-b2ef-7ea43062c57f-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:11.428610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.428602    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:11.428610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.428612    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:11.428868 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.428623    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4w8s\" (UniqueName: \"kubernetes.io/projected/4c738099-89b4-4180-b2ef-7ea43062c57f-kube-api-access-z4w8s\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:11.428868 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:11.428632    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c738099-89b4-4180-b2ef-7ea43062c57f-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:12.038330 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.038294    4528 generic.go:356] "Generic (PLEG): container finished" podID="4c738099-89b4-4180-b2ef-7ea43062c57f" containerID="2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764" exitCode=0
Oct 30 03:45:12.038853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.038338    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.038853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.038373    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c738099-89b4-4180-b2ef-7ea43062c57f","Type":"ContainerDied","Data":"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"}
Oct 30 03:45:12.038853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.038427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4c738099-89b4-4180-b2ef-7ea43062c57f","Type":"ContainerDied","Data":"bf62fdfbaa438746296e9cc1e2fe9bc6d7c595b43bff683d937eb8bd69e19f3b"}
Oct 30 03:45:12.038853 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.038449    4528 scope.go:122] "RemoveContainer" containerID="2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"
Oct 30 03:45:12.046531 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.046510    4528 scope.go:122] "RemoveContainer" containerID="2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"
Oct 30 03:45:12.046925 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:45:12.046800    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764\": container with ID starting with 2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764 not found: ID does not exist" containerID="2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"
Oct 30 03:45:12.047044 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.046944    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764"} err="failed to get container status \"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764\": rpc error: code = NotFound desc = could not find container \"2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764\": container with ID starting with 2fbecda552a4ff0aba9696a0dbc310910504b17fc480ec26eaa0cad8606b9764 not found: ID does not exist"
Oct 30 03:45:12.082082 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.082052    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:12.091592 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.091568    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:12.131925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.131891    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:12.132278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.132251    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4c738099-89b4-4180-b2ef-7ea43062c57f" containerName="otc-container"
Oct 30 03:45:12.132278 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.132269    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c738099-89b4-4180-b2ef-7ea43062c57f" containerName="otc-container"
Oct 30 03:45:12.132410 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.132351    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c738099-89b4-4180-b2ef-7ea43062c57f" containerName="otc-container"
Oct 30 03:45:12.136813 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.136794    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.149929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.149866    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:45:12.149929 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.149924    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:45:12.150057 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.149952    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:45:12.150057 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.150044    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:45:12.150124 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.150045    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:45:12.160580 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.160559    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:12.235995 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.235960    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.236181 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.236012    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.236250 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.236176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfxb\" (UniqueName: \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.236250 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.236221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.236355 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.236305    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.337696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.337659    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfxb\" (UniqueName: \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.337696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.337696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.337971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.337721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.337971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.337743    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.337971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.337770    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.338509 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.338482    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.340279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.340249    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.340391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.340369    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.340443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.340370    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.358267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.358242    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfxb\" (UniqueName: \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\") pod \"data-science-collector-collector-1\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.449448 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.449370    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:12.745396 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.745316    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:12.750440 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:45:12.750409    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de0ade1_a955_4c2c_9dba_45975ae54b47.slice/crio-1703fe30f7af4300624154d593fd300b33cefac39369a1827d64d6f709a7d94f WatchSource:0}: Error finding container 1703fe30f7af4300624154d593fd300b33cefac39369a1827d64d6f709a7d94f: Status 404 returned error can't find the container with id 1703fe30f7af4300624154d593fd300b33cefac39369a1827d64d6f709a7d94f
Oct 30 03:45:12.752573 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.752543    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:12.752630 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:12.752602    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:13.044108 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:13.044012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5de0ade1-a955-4c2c-9dba-45975ae54b47","Type":"ContainerStarted","Data":"af08a83af885290c91d38dd79796854a1a4adeb14a3b548f8d7d97efa9eba54c"}
Oct 30 03:45:13.044108 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:13.044077    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5de0ade1-a955-4c2c-9dba-45975ae54b47","Type":"ContainerStarted","Data":"1703fe30f7af4300624154d593fd300b33cefac39369a1827d64d6f709a7d94f"}
Oct 30 03:45:13.082745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:13.082694    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.082680774 podStartE2EDuration="1.082680774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:13.081631793 +0000 UTC m=+3871.305881070" watchObservedRunningTime="2026-10-30 03:45:13.082680774 +0000 UTC m=+3871.306930050"
Oct 30 03:45:13.892664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:13.892629    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c738099-89b4-4180-b2ef-7ea43062c57f" path="/var/lib/kubelet/pods/4c738099-89b4-4180-b2ef-7ea43062c57f/volumes"
Oct 30 03:45:13.935731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:13.935700    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:14.079740 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:14.079708    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:14.598750 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:14.598713    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:20.676948 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:20.676913    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:20.803112 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:20.803066    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:25.024010 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:25.023964    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:25.148138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:25.148097    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:25.262181 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:25.262146    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:25.385142 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:25.385109    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:26.919625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:26.919586    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:26.920161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:26.919794    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5de0ade1-a955-4c2c-9dba-45975ae54b47" containerName="otc-container" containerID="cri-o://af08a83af885290c91d38dd79796854a1a4adeb14a3b548f8d7d97efa9eba54c" gracePeriod=30
Oct 30 03:45:27.080415 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:27.080365    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.197323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:27.197240    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:27.320585 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:27.320544    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:28.110829 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.110786    4528 generic.go:356] "Generic (PLEG): container finished" podID="5de0ade1-a955-4c2c-9dba-45975ae54b47" containerID="af08a83af885290c91d38dd79796854a1a4adeb14a3b548f8d7d97efa9eba54c" exitCode=0
Oct 30 03:45:28.111187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.110852    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5de0ade1-a955-4c2c-9dba-45975ae54b47","Type":"ContainerDied","Data":"af08a83af885290c91d38dd79796854a1a4adeb14a3b548f8d7d97efa9eba54c"}
Oct 30 03:45:28.163308 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.163285    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:28.273502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273411    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\" (UniqueName: \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\") pod \"5de0ade1-a955-4c2c-9dba-45975ae54b47\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") "
Oct 30 03:45:28.273502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273480    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\") pod \"5de0ade1-a955-4c2c-9dba-45975ae54b47\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") "
Oct 30 03:45:28.273731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273520    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\") pod \"5de0ade1-a955-4c2c-9dba-45975ae54b47\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") "
Oct 30 03:45:28.273731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273564    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\") pod \"5de0ade1-a955-4c2c-9dba-45975ae54b47\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") "
Oct 30 03:45:28.273731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273596    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\") pod \"5de0ade1-a955-4c2c-9dba-45975ae54b47\" (UID: \"5de0ade1-a955-4c2c-9dba-45975ae54b47\") "
Oct 30 03:45:28.273959 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.273936    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal" pod "5de0ade1-a955-4c2c-9dba-45975ae54b47" (UID: "5de0ade1-a955-4c2c-9dba-45975ae54b47"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:28.275726 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.275685    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs" pod "5de0ade1-a955-4c2c-9dba-45975ae54b47" (UID: "5de0ade1-a955-4c2c-9dba-45975ae54b47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:28.275859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.275737    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert" pod "5de0ade1-a955-4c2c-9dba-45975ae54b47" (UID: "5de0ade1-a955-4c2c-9dba-45975ae54b47"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:28.275859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.275763    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb" pod "5de0ade1-a955-4c2c-9dba-45975ae54b47" (UID: "5de0ade1-a955-4c2c-9dba-45975ae54b47"). InnerVolumeSpecName "kube-api-access-5mfxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:28.275859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.275790    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs" pod "5de0ade1-a955-4c2c-9dba-45975ae54b47" (UID: "5de0ade1-a955-4c2c-9dba-45975ae54b47"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:28.374636 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.374596    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5de0ade1-a955-4c2c-9dba-45975ae54b47-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.374636 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.374630    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.374636 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.374639    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.374908 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.374651    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mfxb\" (UniqueName: \"kubernetes.io/projected/5de0ade1-a955-4c2c-9dba-45975ae54b47-kube-api-access-5mfxb\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.374908 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.374661    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5de0ade1-a955-4c2c-9dba-45975ae54b47-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:45:28.912722 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.912684    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:28.962368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.962324    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:45:28.962829 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.962802    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5de0ade1-a955-4c2c-9dba-45975ae54b47" containerName="otc-container"
Oct 30 03:45:28.962924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.962838    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de0ade1-a955-4c2c-9dba-45975ae54b47" containerName="otc-container"
Oct 30 03:45:28.962959 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.962938    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de0ade1-a955-4c2c-9dba-45975ae54b47" containerName="otc-container"
Oct 30 03:45:28.966417 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.966399    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:28.979337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.979311    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"openshift-service-ca.crt\""
Oct 30 03:45:28.988878 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.988858    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"tests-monitoring-tenant-metrics\"/\"kube-root-ca.crt\""
Oct 30 03:45:28.988990 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.988881    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"tests-monitoring-tenant-metrics\"/\"default-dockercfg-mlgkh\""
Oct 30 03:45:28.990283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:28.990264    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:45:29.045198 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.045163    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:29.081913 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.081880    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.081913 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.081916    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcb2\" (UniqueName: \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.116767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.116735    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.116767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.116764    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5de0ade1-a955-4c2c-9dba-45975ae54b47","Type":"ContainerDied","Data":"1703fe30f7af4300624154d593fd300b33cefac39369a1827d64d6f709a7d94f"}
Oct 30 03:45:29.117282 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.116814    4528 scope.go:122] "RemoveContainer" containerID="af08a83af885290c91d38dd79796854a1a4adeb14a3b548f8d7d97efa9eba54c"
Oct 30 03:45:29.161385 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.161351    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:29.163384 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.163324    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:29.167149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.167121    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:45:29.183143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.183116    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.183229 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.183150    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcb2\" (UniqueName: \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.183498 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.183480    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.203063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.203028    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:29.204514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.204490    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcb2\" (UniqueName: \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\") pod \"tenant-metrics-exporter-5cf544fb97-hx9q7\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") " pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.207854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.207839    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.220879 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.220861    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:45:29.221029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.221016    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:45:29.223295 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.223268    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:45:29.223295 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.223269    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:45:29.223425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.223409    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:45:29.226654 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.226636    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:29.279361 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.279324    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:45:29.385192 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.385156    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.385355 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.385205    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvql7\" (UniqueName: \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.385355 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.385281    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.385355 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.385330    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.385452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.385361    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486052 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.485982    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486052 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.486034    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvql7\" (UniqueName: \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.486065    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.486106    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.486137    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.486912 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.486882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.488531 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.488506    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.488618 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.488584    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.488685 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.488658    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.509183 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.509164    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvql7\" (UniqueName: \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\") pod \"data-science-collector-collector-1\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.522513 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.522493    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:45:29.576391 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.576306    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:45:29.589282 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:45:29.589257    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54c1aa7_da01_4266_8cea_71f80b26ce43.slice/crio-14b7bda2e749dd5a2a82402ac1a783b83e27c40ac4c9a8ff2dbf7c21729c13b0 WatchSource:0}: Error finding container 14b7bda2e749dd5a2a82402ac1a783b83e27c40ac4c9a8ff2dbf7c21729c13b0: Status 404 returned error can't find the container with id 14b7bda2e749dd5a2a82402ac1a783b83e27c40ac4c9a8ff2dbf7c21729c13b0
Oct 30 03:45:29.591051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.591033    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:45:29.820879 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.820787    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:45:29.825436 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:45:29.825404    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dead43_d0ba_4a03_ba42_5d826e3134de.slice/crio-01fcbc8d4230e8af0135f3d7a76517f5a270b66f98cc5174e4c759b4f82c2aef WatchSource:0}: Error finding container 01fcbc8d4230e8af0135f3d7a76517f5a270b66f98cc5174e4c759b4f82c2aef: Status 404 returned error can't find the container with id 01fcbc8d4230e8af0135f3d7a76517f5a270b66f98cc5174e4c759b4f82c2aef
Oct 30 03:45:29.827587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.827556    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:29.827685 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.827626    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:45:29.893210 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:29.893176    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de0ade1-a955-4c2c-9dba-45975ae54b47" path="/var/lib/kubelet/pods/5de0ade1-a955-4c2c-9dba-45975ae54b47/volumes"
Oct 30 03:45:30.134951 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:30.134904    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" event={"ID":"d54c1aa7-da01-4266-8cea-71f80b26ce43","Type":"ContainerStarted","Data":"14b7bda2e749dd5a2a82402ac1a783b83e27c40ac4c9a8ff2dbf7c21729c13b0"}
Oct 30 03:45:30.141213 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:30.141178    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"30dead43-d0ba-4a03-ba42-5d826e3134de","Type":"ContainerStarted","Data":"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"}
Oct 30 03:45:30.141343 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:30.141247    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"30dead43-d0ba-4a03-ba42-5d826e3134de","Type":"ContainerStarted","Data":"01fcbc8d4230e8af0135f3d7a76517f5a270b66f98cc5174e4c759b4f82c2aef"}
Oct 30 03:45:30.183328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:30.183272    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.1832588450000001 podStartE2EDuration="1.183258845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:30.182926245 +0000 UTC m=+3888.407175521" watchObservedRunningTime="2026-10-30 03:45:30.183258845 +0000 UTC m=+3888.407508116"
Oct 30 03:45:33.156716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:33.156631    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" event={"ID":"d54c1aa7-da01-4266-8cea-71f80b26ce43","Type":"ContainerStarted","Data":"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"}
Oct 30 03:45:33.191248 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:33.191191    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" podStartSLOduration=1.8980169180000002 podStartE2EDuration="5.19117629s" totalImagesPullingTime="3.293159372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:33.191138134 +0000 UTC m=+3891.415387412" watchObservedRunningTime="2026-10-30 03:45:33.19117629 +0000 UTC m=+3891.415425563"
Oct 30 03:45:34.630552 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:45:34.630512    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:45:37.587423 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:45:37.587383    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:04.630390 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:46:04.630311    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:46:18.289314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:18.289276    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:20.894751 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:20.894711    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:46:20.895419 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:20.895022    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" podUID="d54c1aa7-da01-4266-8cea-71f80b26ce43" containerName="tenant-metrics-exporter" containerID="cri-o://4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5" gracePeriod=30
Oct 30 03:46:20.995339 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:20.995302    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:21.111184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.111138    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:21.147651 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.147589    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:46:21.179539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.179509    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\" (UniqueName: \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\") pod \"d54c1aa7-da01-4266-8cea-71f80b26ce43\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") "
Oct 30 03:46:21.179693 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.179552    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\" (UniqueName: \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\") pod \"d54c1aa7-da01-4266-8cea-71f80b26ce43\" (UID: \"d54c1aa7-da01-4266-8cea-71f80b26ce43\") "
Oct 30 03:46:21.180015 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.179993    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data" pod "d54c1aa7-da01-4266-8cea-71f80b26ce43" (UID: "d54c1aa7-da01-4266-8cea-71f80b26ce43"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:21.181695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.181675    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2" pod "d54c1aa7-da01-4266-8cea-71f80b26ce43" (UID: "d54c1aa7-da01-4266-8cea-71f80b26ce43"). InnerVolumeSpecName "kube-api-access-ppcb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:21.280311 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.280273    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcb2\" (UniqueName: \"kubernetes.io/projected/d54c1aa7-da01-4266-8cea-71f80b26ce43-kube-api-access-ppcb2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:46:21.280311 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.280301    4528 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d54c1aa7-da01-4266-8cea-71f80b26ce43-data\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:46:21.362279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.362245    4528 generic.go:356] "Generic (PLEG): container finished" podID="d54c1aa7-da01-4266-8cea-71f80b26ce43" containerID="4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5" exitCode=0
Oct 30 03:46:21.362487 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.362297    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"
Oct 30 03:46:21.362487 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.362320    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" event={"ID":"d54c1aa7-da01-4266-8cea-71f80b26ce43","Type":"ContainerDied","Data":"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"}
Oct 30 03:46:21.362487 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.362363    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7" event={"ID":"d54c1aa7-da01-4266-8cea-71f80b26ce43","Type":"ContainerDied","Data":"14b7bda2e749dd5a2a82402ac1a783b83e27c40ac4c9a8ff2dbf7c21729c13b0"}
Oct 30 03:46:21.362487 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.362383    4528 scope.go:122] "RemoveContainer" containerID="4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"
Oct 30 03:46:21.372358 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.372336    4528 scope.go:122] "RemoveContainer" containerID="4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"
Oct 30 03:46:21.372634 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:46:21.372615    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5\": container with ID starting with 4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5 not found: ID does not exist" containerID="4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"
Oct 30 03:46:21.372703 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.372642    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5"} err="failed to get container status \"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5\": rpc error: code = NotFound desc = could not find container \"4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5\": container with ID starting with 4eb9b0d02a3ae4e1b05a73918947d4632c0fcee8e66184e595734e92a0b8b5f5 not found: ID does not exist"
Oct 30 03:46:21.405586 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.405513    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:46:21.412321 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.412291    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["tests-monitoring-tenant-metrics/tenant-metrics-exporter-5cf544fb97-hx9q7"]
Oct 30 03:46:21.891881 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:21.891851    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54c1aa7-da01-4266-8cea-71f80b26ce43" path="/var/lib/kubelet/pods/d54c1aa7-da01-4266-8cea-71f80b26ce43/volumes"
Oct 30 03:46:26.961160 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:26.961122    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:27.087272 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:27.087236    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:27.207217 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:27.207187    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:32.923057 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:32.923001    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:33.040216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:33.040167    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:33.411814 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:33.411775    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:33.529249 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:33.529204    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:33.655075 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:33.655035    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:33.774547 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:33.774446    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:46:34.629942 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:46:34.629900    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:46:37.715348 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:46:37.715308    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:47:04.631362 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:47:04.631325    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:47:34.631220 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:47:34.631180    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:47:37.848677 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:47:37.848632    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:04.630840 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:48:04.630789    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:48:18.483866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:48:18.483808    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:23.111042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:48:23.111003    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:23.249646 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:48:23.249611    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:48:34.631811 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:48:34.631776    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:48:37.972692 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:48:37.972647    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:04.632831 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:49:04.632736    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:49:21.600656 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:49:21.600612    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:49:34.630341 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:49:34.630301    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:49:38.094526 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:49:38.094496    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:04.631117 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:50:04.631082    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:50:34.078161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:34.078105    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:34.631410 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:50:34.631376    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:50:38.212015 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:38.211980    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:39.748187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:39.748154    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:39.865496 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:39.865455    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:40.224552 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:40.224496    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:40.354020 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:40.353981    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:46.071340 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:46.071302    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:46.209187 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:46.209147    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:46.369930 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:46.369836    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:52.186687 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:52.186650    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:52.300985 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:52.300949    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:57.164429 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:57.164386    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:57.365695 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:57.365658    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:57.484111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:57.484034    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.053051 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:59.053012    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:50:59.053556 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:59.053311    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="30dead43-d0ba-4a03-ba42-5d826e3134de" containerName="otc-container" containerID="cri-o://86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036" gracePeriod=30
Oct 30 03:50:59.204854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:59.204799    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.319404 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:59.319319    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:50:59.435866 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:50:59.435806    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:01.391224 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.391199    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.435453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435375    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\" (UniqueName: \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\") pod \"30dead43-d0ba-4a03-ba42-5d826e3134de\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") "
Oct 30 03:51:01.435453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435412    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\") pod \"30dead43-d0ba-4a03-ba42-5d826e3134de\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") "
Oct 30 03:51:01.435658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435450    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\") pod \"30dead43-d0ba-4a03-ba42-5d826e3134de\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") "
Oct 30 03:51:01.435658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435502    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\" (UniqueName: \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\") pod \"30dead43-d0ba-4a03-ba42-5d826e3134de\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") "
Oct 30 03:51:01.435658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435548    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\") pod \"30dead43-d0ba-4a03-ba42-5d826e3134de\" (UID: \"30dead43-d0ba-4a03-ba42-5d826e3134de\") "
Oct 30 03:51:01.435943 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.435916    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal" pod "30dead43-d0ba-4a03-ba42-5d826e3134de" (UID: "30dead43-d0ba-4a03-ba42-5d826e3134de"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:51:01.437890 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.437783    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7" pod "30dead43-d0ba-4a03-ba42-5d826e3134de" (UID: "30dead43-d0ba-4a03-ba42-5d826e3134de"). InnerVolumeSpecName "kube-api-access-rvql7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:51:01.438147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.438121    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs" pod "30dead43-d0ba-4a03-ba42-5d826e3134de" (UID: "30dead43-d0ba-4a03-ba42-5d826e3134de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:01.438379 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.438350    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert" pod "30dead43-d0ba-4a03-ba42-5d826e3134de" (UID: "30dead43-d0ba-4a03-ba42-5d826e3134de"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:01.438493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.438419    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs" pod "30dead43-d0ba-4a03-ba42-5d826e3134de" (UID: "30dead43-d0ba-4a03-ba42-5d826e3134de"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:51:01.497446 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.497414    4528 generic.go:356] "Generic (PLEG): container finished" podID="30dead43-d0ba-4a03-ba42-5d826e3134de" containerID="86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036" exitCode=0
Oct 30 03:51:01.497600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.497464    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.497600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.497478    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"30dead43-d0ba-4a03-ba42-5d826e3134de","Type":"ContainerDied","Data":"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"}
Oct 30 03:51:01.497600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.497519    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"30dead43-d0ba-4a03-ba42-5d826e3134de","Type":"ContainerDied","Data":"01fcbc8d4230e8af0135f3d7a76517f5a270b66f98cc5174e4c759b4f82c2aef"}
Oct 30 03:51:01.497600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.497539    4528 scope.go:122] "RemoveContainer" containerID="86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"
Oct 30 03:51:01.506452 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.506430    4528 scope.go:122] "RemoveContainer" containerID="86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"
Oct 30 03:51:01.506707 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:51:01.506685    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036\": container with ID starting with 86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036 not found: ID does not exist" containerID="86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"
Oct 30 03:51:01.506767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.506715    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036"} err="failed to get container status \"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036\": rpc error: code = NotFound desc = could not find container \"86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036\": container with ID starting with 86618d46da03f871951d8dd55560e47316edf3b4aab6c197b05a6fe8ae0ff036 not found: ID does not exist"
Oct 30 03:51:01.536174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.536147    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:51:01.536174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.536173    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30dead43-d0ba-4a03-ba42-5d826e3134de-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:51:01.536357 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.536184    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:51:01.536357 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.536193    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvql7\" (UniqueName: \"kubernetes.io/projected/30dead43-d0ba-4a03-ba42-5d826e3134de-kube-api-access-rvql7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:51:01.536357 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.536202    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30dead43-d0ba-4a03-ba42-5d826e3134de-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:51:01.541170 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.541143    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:51:01.550085 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.550060    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:51:01.597028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597002    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:51:01.597377 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597362    4528 cpu_manager.go:414] "containerMap: removing container" podUID="30dead43-d0ba-4a03-ba42-5d826e3134de" containerName="otc-container"
Oct 30 03:51:01.597447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597379    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dead43-d0ba-4a03-ba42-5d826e3134de" containerName="otc-container"
Oct 30 03:51:01.597447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597393    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d54c1aa7-da01-4266-8cea-71f80b26ce43" containerName="tenant-metrics-exporter"
Oct 30 03:51:01.597447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597399    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54c1aa7-da01-4266-8cea-71f80b26ce43" containerName="tenant-metrics-exporter"
Oct 30 03:51:01.597573 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597459    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dead43-d0ba-4a03-ba42-5d826e3134de" containerName="otc-container"
Oct 30 03:51:01.597573 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.597472    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54c1aa7-da01-4266-8cea-71f80b26ce43" containerName="tenant-metrics-exporter"
Oct 30 03:51:01.601720 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.601701    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.615256 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.615229    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-dkphb\""
Oct 30 03:51:01.615365 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.615299    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 03:51:01.615365 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.615313    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:51:01.615447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.615369    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 03:51:01.615447 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.615392    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 03:51:01.622632 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.622611    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:51:01.636965 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.636936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.637102 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.636982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99sdm\" (UniqueName: \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.637102 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.637074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.637102 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.637096    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.637283 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.637132    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.738264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.738186    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.738264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.738224    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.738264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.738254    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.738524 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.738297    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.738524 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.738334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99sdm\" (UniqueName: \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.739060 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.739034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.740661 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.740639    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.740763 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.740669    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.740763 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.740700    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.759533 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.759514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99sdm\" (UniqueName: \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\") pod \"data-science-collector-collector-1\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:01.892756 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.892726    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30dead43-d0ba-4a03-ba42-5d826e3134de" path="/var/lib/kubelet/pods/30dead43-d0ba-4a03-ba42-5d826e3134de/volumes"
Oct 30 03:51:01.913976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:01.913950    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:51:02.213108 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.213075    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:51:02.218127 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:51:02.218100    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b6e223_8336_450b_b67b_27576e554cd3.slice/crio-1c5fc55e9296f0974ecafa6096fb18022a3e64791db51ce11332d06c37ea2f8b WatchSource:0}: Error finding container 1c5fc55e9296f0974ecafa6096fb18022a3e64791db51ce11332d06c37ea2f8b: Status 404 returned error can't find the container with id 1c5fc55e9296f0974ecafa6096fb18022a3e64791db51ce11332d06c37ea2f8b
Oct 30 03:51:02.220161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.220130    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:51:02.220240 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.220189    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:51:02.505059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.504971    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b0b6e223-8336-450b-b67b-27576e554cd3","Type":"ContainerStarted","Data":"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"}
Oct 30 03:51:02.505059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.505034    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b0b6e223-8336-450b-b67b-27576e554cd3","Type":"ContainerStarted","Data":"1c5fc55e9296f0974ecafa6096fb18022a3e64791db51ce11332d06c37ea2f8b"}
Oct 30 03:51:02.543793 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:02.543727    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.543710367 podStartE2EDuration="1.543710367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.542940414 +0000 UTC m=+4220.767189691" watchObservedRunningTime="2026-10-30 03:51:02.543710367 +0000 UTC m=+4220.767959645"
Oct 30 03:51:04.634695 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:51:04.634651    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:51:22.073476 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.073441    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"]
Oct 30 03:51:22.076871 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.076849    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"
Oct 30 03:51:22.089728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.089706    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 03:51:22.097581 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.097553    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"
Oct 30 03:51:22.100981 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.100862    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"]
Oct 30 03:51:22.392177 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.392148    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"]
Oct 30 03:51:22.397068 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:51:22.397037    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd17f75f1_c2aa_4dfa_8f3f_245b30f839bd.slice/crio-6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744 WatchSource:0}: Error finding container 6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744: Status 404 returned error can't find the container with id 6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744
Oct 30 03:51:22.583300 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.583257    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae" event={"ID":"d17f75f1-c2aa-4dfa-8f3f-245b30f839bd","Type":"ContainerStarted","Data":"d8037d242b997cee58212a6d1ff27047ee657c6eb2511263c3f57171a41abd24"}
Oct 30 03:51:22.583485 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.583309    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae" event={"ID":"d17f75f1-c2aa-4dfa-8f3f-245b30f839bd","Type":"ContainerStarted","Data":"6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744"}
Oct 30 03:51:22.617838 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:22.617767    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae" podStartSLOduration=0.617753094 podStartE2EDuration="617.753094ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:22.616873702 +0000 UTC m=+4240.841122984" watchObservedRunningTime="2026-10-30 03:51:22.617753094 +0000 UTC m=+4240.842002386"
Oct 30 03:51:32.626585 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:32.626549    4528 generic.go:356] "Generic (PLEG): container finished" podID="d17f75f1-c2aa-4dfa-8f3f-245b30f839bd" containerID="d8037d242b997cee58212a6d1ff27047ee657c6eb2511263c3f57171a41abd24" exitCode=0
Oct 30 03:51:32.626997 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:32.626587    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae" event={"ID":"d17f75f1-c2aa-4dfa-8f3f-245b30f839bd","Type":"ContainerDied","Data":"d8037d242b997cee58212a6d1ff27047ee657c6eb2511263c3f57171a41abd24"}
Oct 30 03:51:33.758349 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:33.758326    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"
Oct 30 03:51:34.314549 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.314471    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:34.383659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.383623    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"]
Oct 30 03:51:34.384103 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.384083    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d17f75f1-c2aa-4dfa-8f3f-245b30f839bd" containerName="curl"
Oct 30 03:51:34.384200 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.384104    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17f75f1-c2aa-4dfa-8f3f-245b30f839bd" containerName="curl"
Oct 30 03:51:34.384200 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.384186    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17f75f1-c2aa-4dfa-8f3f-245b30f839bd" containerName="curl"
Oct 30 03:51:34.387078 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.387057    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"
Oct 30 03:51:34.397647 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.397628    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"
Oct 30 03:51:34.408731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.408708    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"]
Oct 30 03:51:34.430413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.430368    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:34.631513 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:51:34.631479    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:51:34.635712 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.635685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae" event={"ID":"d17f75f1-c2aa-4dfa-8f3f-245b30f839bd","Type":"ContainerDied","Data":"6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744"}
Oct 30 03:51:34.635903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.635724    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a1d847024f567410b03da229295337eb9f2e56b67b1ac0d67fa4371d3c7d744"
Oct 30 03:51:34.635903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.635692    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"
Oct 30 03:51:34.702298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:34.702261    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"]
Oct 30 03:51:34.707007 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:51:34.706979    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29bd6358_a77c_4265_b6cf_3cd52eedb6ea.slice/crio-6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e WatchSource:0}: Error finding container 6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e: Status 404 returned error can't find the container with id 6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e
Oct 30 03:51:35.641107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:35.641073    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e" event={"ID":"29bd6358-a77c-4265-b6cf-3cd52eedb6ea","Type":"ContainerStarted","Data":"7da3d557559e9bf650f30f239edd2691fffca7fbaf79f09d586a6a252145f915"}
Oct 30 03:51:35.641107 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:35.641112    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e" event={"ID":"29bd6358-a77c-4265-b6cf-3cd52eedb6ea","Type":"ContainerStarted","Data":"6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e"}
Oct 30 03:51:35.677834 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:35.677780    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e" podStartSLOduration=1.677767046 podStartE2EDuration="1.677767046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:35.677339298 +0000 UTC m=+4253.901588577" watchObservedRunningTime="2026-10-30 03:51:35.677767046 +0000 UTC m=+4253.902016320"
Oct 30 03:51:38.333176 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:38.333120    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:51:45.690635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:45.690597    4528 generic.go:356] "Generic (PLEG): container finished" podID="29bd6358-a77c-4265-b6cf-3cd52eedb6ea" containerID="7da3d557559e9bf650f30f239edd2691fffca7fbaf79f09d586a6a252145f915" exitCode=0
Oct 30 03:51:45.691021 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:45.690641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e" event={"ID":"29bd6358-a77c-4265-b6cf-3cd52eedb6ea","Type":"ContainerDied","Data":"7da3d557559e9bf650f30f239edd2691fffca7fbaf79f09d586a6a252145f915"}
Oct 30 03:51:46.823408 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:46.823381    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"
Oct 30 03:51:47.700006 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:47.699973    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"
Oct 30 03:51:47.700006 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:47.700000    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e" event={"ID":"29bd6358-a77c-4265-b6cf-3cd52eedb6ea","Type":"ContainerDied","Data":"6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e"}
Oct 30 03:51:47.700216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:51:47.700032    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6597001c4fb8ee4030a34f23a5ca586563fe3d72f1ad9d803df933e8e2db030e"
Oct 30 03:52:04.631795 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:52:04.631711    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:52:14.598261 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:52:14.598222    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:19.622065 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:52:19.622024    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:52:34.630764 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:52:34.630724    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:52:38.483880 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:52:38.483841    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:53:04.631189 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:53:04.631145    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:53:34.630580 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:53:34.630529    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:53:38.605028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:53:38.604994    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:54:04.631341 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:54:04.631302    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:54:34.631529 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:54:34.631491    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:54:38.716995 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:54:38.716946    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:04.631276 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:55:04.631193    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:55:19.800837 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:19.800775    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:34.631630 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:55:34.631595    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:55:37.803215 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.803135    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"]
Oct 30 03:55:37.807517 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.807490    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-otlp-workload-allowed-18e332385aa9bf2e"]
Oct 30 03:55:37.846545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.846515    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"]
Oct 30 03:55:37.850374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.850352    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-untrusted-denied-18e332357f7955ae"]
Oct 30 03:55:37.892626 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.892594    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bd6358-a77c-4265-b6cf-3cd52eedb6ea" path="/var/lib/kubelet/pods/29bd6358-a77c-4265-b6cf-3cd52eedb6ea/volumes"
Oct 30 03:55:37.892900 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.892885    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17f75f1-c2aa-4dfa-8f3f-245b30f839bd" path="/var/lib/kubelet/pods/d17f75f1-c2aa-4dfa-8f3f-245b30f839bd/volumes"
Oct 30 03:55:37.929793 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:37.929757    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:38.048767 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:38.048728    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:38.839444 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:38.839412    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:43.880150 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:43.880111    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:43.995296 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:43.995262    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:51.243262 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:51.243224    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:51.405004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:51.404966    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:55:53.699364 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:55:53.699317    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:04.631015 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:04.630970    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:56:09.136081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:09.136046    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:11.618992 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.618953    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:56:11.619606 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.619243    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="kube-rbac-proxy" containerID="cri-o://2ab2e718fcf3c6cbc92b5e9d9707f343bf5ef553445a22786a50e19fd4216d0b" gracePeriod=30
Oct 30 03:56:11.619606 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.619333    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="prom-label-proxy" containerID="cri-o://ca9d6b3dc4f490e811e14e83c293a54994663056d365c1adc2d8ff3d73831328" gracePeriod=30
Oct 30 03:56:11.642917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.642791    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:56:11.643063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.643041    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="kube-rbac-proxy" containerID="cri-o://ccec5e1207bae82737f2edb16f5e0e40df987b33ae243d93791b220c90407bb6" gracePeriod=30
Oct 30 03:56:11.643138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.643043    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="prom-label-proxy" containerID="cri-o://e5a7a90ae9bbe6f486af773849bbf3c7dc48c634752c0dc47eff79177957418d" gracePeriod=30
Oct 30 03:56:11.795132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.794939    4528 generic.go:356] "Generic (PLEG): container finished" podID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerID="ca9d6b3dc4f490e811e14e83c293a54994663056d365c1adc2d8ff3d73831328" exitCode=0
Oct 30 03:56:11.795132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.794964    4528 generic.go:356] "Generic (PLEG): container finished" podID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerID="2ab2e718fcf3c6cbc92b5e9d9707f343bf5ef553445a22786a50e19fd4216d0b" exitCode=0
Oct 30 03:56:11.795132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.795032    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerDied","Data":"ca9d6b3dc4f490e811e14e83c293a54994663056d365c1adc2d8ff3d73831328"}
Oct 30 03:56:11.795132 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.795078    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerDied","Data":"2ab2e718fcf3c6cbc92b5e9d9707f343bf5ef553445a22786a50e19fd4216d0b"}
Oct 30 03:56:11.797056 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.796911    4528 generic.go:356] "Generic (PLEG): container finished" podID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerID="e5a7a90ae9bbe6f486af773849bbf3c7dc48c634752c0dc47eff79177957418d" exitCode=0
Oct 30 03:56:11.797056 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.796931    4528 generic.go:356] "Generic (PLEG): container finished" podID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerID="ccec5e1207bae82737f2edb16f5e0e40df987b33ae243d93791b220c90407bb6" exitCode=0
Oct 30 03:56:11.797056 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.796993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerDied","Data":"e5a7a90ae9bbe6f486af773849bbf3c7dc48c634752c0dc47eff79177957418d"}
Oct 30 03:56:11.797056 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.797025    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerDied","Data":"ccec5e1207bae82737f2edb16f5e0e40df987b33ae243d93791b220c90407bb6"}
Oct 30 03:56:11.900453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.900431    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:56:11.903174 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.903147    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:56:11.912635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.912611    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\" (UniqueName: \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\") pod \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") "
Oct 30 03:56:11.912747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.912646    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\" (UniqueName: \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\") pod \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") "
Oct 30 03:56:11.912747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.912677    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\") pod \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") "
Oct 30 03:56:11.912747 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.912740    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\") pod \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") "
Oct 30 03:56:11.912971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.912936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\" (UniqueName: \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\") pod \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") "
Oct 30 03:56:11.913047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913010    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\") pod \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") "
Oct 30 03:56:11.913111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913069    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\" (UniqueName: \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\") pod \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\" (UID: \"6aeedf29-24be-4352-ac34-e1bb63cf8dc7\") "
Oct 30 03:56:11.913111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca" pod "6aeedf29-24be-4352-ac34-e1bb63cf8dc7" (UID: "6aeedf29-24be-4352-ac34-e1bb63cf8dc7"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:11.913111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913090    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config" pod "6aeedf29-24be-4352-ac34-e1bb63cf8dc7" (UID: "6aeedf29-24be-4352-ac34-e1bb63cf8dc7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:11.913111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913102    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\" (UniqueName: \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\") pod \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\" (UID: \"5b687f0b-239a-4fad-95ee-51a0af2ea98a\") "
Oct 30 03:56:11.913613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913360    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-prometheus-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:11.913613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913378    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:11.913613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca" pod "6aeedf29-24be-4352-ac34-e1bb63cf8dc7" (UID: "6aeedf29-24be-4352-ac34-e1bb63cf8dc7"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:11.913613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.913456    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config" pod "5b687f0b-239a-4fad-95ee-51a0af2ea98a" (UID: "5b687f0b-239a-4fad-95ee-51a0af2ea98a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:11.915025 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.915001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5" pod "5b687f0b-239a-4fad-95ee-51a0af2ea98a" (UID: "5b687f0b-239a-4fad-95ee-51a0af2ea98a"). InnerVolumeSpecName "kube-api-access-8npx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:11.915224 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.915204    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t" pod "6aeedf29-24be-4352-ac34-e1bb63cf8dc7" (UID: "6aeedf29-24be-4352-ac34-e1bb63cf8dc7"). InnerVolumeSpecName "kube-api-access-cgr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:11.915550 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.915534    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls" pod "5b687f0b-239a-4fad-95ee-51a0af2ea98a" (UID: "5b687f0b-239a-4fad-95ee-51a0af2ea98a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:11.915689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:11.915671    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls" pod "6aeedf29-24be-4352-ac34-e1bb63cf8dc7" (UID: "6aeedf29-24be-4352-ac34-e1bb63cf8dc7"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.014106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014076    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5b687f0b-239a-4fad-95ee-51a0af2ea98a-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.014106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014101    4528 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.014106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014111    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.014106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014118    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5b687f0b-239a-4fad-95ee-51a0af2ea98a-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.014106 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014126    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npx5\" (UniqueName: \"kubernetes.io/projected/5b687f0b-239a-4fad-95ee-51a0af2ea98a-kube-api-access-8npx5\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.014530 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.014135    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr2t\" (UniqueName: \"kubernetes.io/projected/6aeedf29-24be-4352-ac34-e1bb63cf8dc7-kube-api-access-cgr2t\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.095556 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.095502    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.233315 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.233236    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:56:12.233579 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.233552    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="prometheus" containerID="cri-o://05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0" gracePeriod=600
Oct 30 03:56:12.233933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.233889    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="config-reloader" containerID="cri-o://e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9" gracePeriod=600
Oct 30 03:56:12.233933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.233916    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="thanos-sidecar" containerID="cri-o://6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174" gracePeriod=600
Oct 30 03:56:12.235712 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.235686    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.353839 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.353786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.483527 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.483469    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:56:12.519432 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519404    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.519600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519465    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.519600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519496    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.519600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519545    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.519600 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519576    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519638    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519666    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519749    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519801    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519846    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519893    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:12.520004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.519907    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\") pod \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\" (UID: \"ed237b85-bb8d-4c84-94a3-2a0ef65954f4\") "
Oct 30 03:56:12.520458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.520144    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.520458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.520342    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:12.520779 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.520733    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:12.522572 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.522548    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.522774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.522749    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.522915 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.522885    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:12.523191 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.523169    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.523728 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.523705    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:12.523850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.523717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.524025 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.524008    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.524569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.524550    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "kube-api-access-rjk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:12.525359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.525338    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:56:12.526997 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.526971    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.536350 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.536327    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config" pod "ed237b85-bb8d-4c84-94a3-2a0ef65954f4" (UID: "ed237b85-bb8d-4c84-94a3-2a0ef65954f4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:12.620933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620897    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.620933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620927    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.620933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620938    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620947    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620956    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620984    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.620993    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.621002    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.621012    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.621020    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjk4v\" (UniqueName: \"kubernetes.io/projected/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-kube-api-access-rjk4v\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.621029    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.621587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.621040    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed237b85-bb8d-4c84-94a3-2a0ef65954f4-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:12.649854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.644495    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.649854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.645377    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:56:12.722031 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.721992    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 03:56:12.802919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.802832    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"
Oct 30 03:56:12.802919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.802842    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b" event={"ID":"6aeedf29-24be-4352-ac34-e1bb63cf8dc7","Type":"ContainerDied","Data":"2e0fb225b796b64ddd3348ae83dd112fe14c17ace446b3d56db7d4d46983ad3f"}
Oct 30 03:56:12.802919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.802920    4528 scope.go:122] "RemoveContainer" containerID="ca9d6b3dc4f490e811e14e83c293a54994663056d365c1adc2d8ff3d73831328"
Oct 30 03:56:12.804525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.804502    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"
Oct 30 03:56:12.804525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.804516    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m" event={"ID":"5b687f0b-239a-4fad-95ee-51a0af2ea98a","Type":"ContainerDied","Data":"ea955242348ffa13af476b5625ade70b7f124623182ee2d784a4c1eaf3773ebd"}
Oct 30 03:56:12.807723 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807683    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174" exitCode=0
Oct 30 03:56:12.807723 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807722    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9" exitCode=0
Oct 30 03:56:12.807903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807737    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0" exitCode=0
Oct 30 03:56:12.807903 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807775    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerDied","Data":"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"}
Oct 30 03:56:12.808023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807902    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerDied","Data":"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"}
Oct 30 03:56:12.808023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.807959    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerDied","Data":"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"}
Oct 30 03:56:12.808023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.808012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"ed237b85-bb8d-4c84-94a3-2a0ef65954f4","Type":"ContainerDied","Data":"c37146ff79432b75038a410e0463869058dd5af90115f3fab201f068f93dba55"}
Oct 30 03:56:12.808274 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.808258    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 03:56:12.819977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.819948    4528 scope.go:122] "RemoveContainer" containerID="2ab2e718fcf3c6cbc92b5e9d9707f343bf5ef553445a22786a50e19fd4216d0b"
Oct 30 03:56:12.828531 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.828474    4528 scope.go:122] "RemoveContainer" containerID="e5a7a90ae9bbe6f486af773849bbf3c7dc48c634752c0dc47eff79177957418d"
Oct 30 03:56:12.837592 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.837576    4528 scope.go:122] "RemoveContainer" containerID="ccec5e1207bae82737f2edb16f5e0e40df987b33ae243d93791b220c90407bb6"
Oct 30 03:56:12.844564 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.844544    4528 scope.go:122] "RemoveContainer" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"
Oct 30 03:56:12.851356 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.851333    4528 scope.go:122] "RemoveContainer" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"
Oct 30 03:56:12.859323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.859302    4528 scope.go:122] "RemoveContainer" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"
Oct 30 03:56:12.869110 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.869087    4528 scope.go:122] "RemoveContainer" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"
Oct 30 03:56:12.874724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.874699    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:56:12.878643 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.878624    4528 scope.go:122] "RemoveContainer" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"
Oct 30 03:56:12.879048 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:12.879023    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": container with ID starting with 6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174 not found: ID does not exist" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"
Oct 30 03:56:12.879147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879058    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"} err="failed to get container status \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": rpc error: code = NotFound desc = could not find container \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": container with ID starting with 6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174 not found: ID does not exist"
Oct 30 03:56:12.879147 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879082    4528 scope.go:122] "RemoveContainer" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"
Oct 30 03:56:12.879489 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:12.879467    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": container with ID starting with e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9 not found: ID does not exist" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"
Oct 30 03:56:12.879588 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879496    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"} err="failed to get container status \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": rpc error: code = NotFound desc = could not find container \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": container with ID starting with e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9 not found: ID does not exist"
Oct 30 03:56:12.879588 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879516    4528 scope.go:122] "RemoveContainer" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"
Oct 30 03:56:12.879813 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:12.879795    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": container with ID starting with 05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0 not found: ID does not exist" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"
Oct 30 03:56:12.879898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879831    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"} err="failed to get container status \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": rpc error: code = NotFound desc = could not find container \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": container with ID starting with 05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0 not found: ID does not exist"
Oct 30 03:56:12.879898 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.879850    4528 scope.go:122] "RemoveContainer" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"
Oct 30 03:56:12.880158 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:12.880134    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": container with ID starting with 4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b not found: ID does not exist" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"
Oct 30 03:56:12.880265 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.880161    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"} err="failed to get container status \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": rpc error: code = NotFound desc = could not find container \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": container with ID starting with 4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b not found: ID does not exist"
Oct 30 03:56:12.880265 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.880181    4528 scope.go:122] "RemoveContainer" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"
Oct 30 03:56:12.880745 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.880464    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-4bc5b"]
Oct 30 03:56:12.881626 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.881602    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"} err="failed to get container status \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": rpc error: code = NotFound desc = could not find container \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": container with ID starting with 6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174 not found: ID does not exist"
Oct 30 03:56:12.881708 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.881626    4528 scope.go:122] "RemoveContainer" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"
Oct 30 03:56:12.881947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.881924    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"} err="failed to get container status \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": rpc error: code = NotFound desc = could not find container \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": container with ID starting with e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9 not found: ID does not exist"
Oct 30 03:56:12.881947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.881946    4528 scope.go:122] "RemoveContainer" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"
Oct 30 03:56:12.882257 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882224    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"} err="failed to get container status \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": rpc error: code = NotFound desc = could not find container \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": container with ID starting with 05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0 not found: ID does not exist"
Oct 30 03:56:12.882362 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882257    4528 scope.go:122] "RemoveContainer" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"
Oct 30 03:56:12.882557 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882531    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"} err="failed to get container status \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": rpc error: code = NotFound desc = could not find container \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": container with ID starting with 4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b not found: ID does not exist"
Oct 30 03:56:12.882613 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882558    4528 scope.go:122] "RemoveContainer" containerID="6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"
Oct 30 03:56:12.882841 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882800    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174"} err="failed to get container status \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": rpc error: code = NotFound desc = could not find container \"6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174\": container with ID starting with 6419da82a03206fb116f6b9da4214ae85bbb0319c9b098b050e146909b146174 not found: ID does not exist"
Oct 30 03:56:12.882946 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.882843    4528 scope.go:122] "RemoveContainer" containerID="e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"
Oct 30 03:56:12.883184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.883148    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9"} err="failed to get container status \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": rpc error: code = NotFound desc = could not find container \"e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9\": container with ID starting with e64a5b04715247eba561413ba602e411f4109d3df3a1e77c7b5f5dc2ae5867a9 not found: ID does not exist"
Oct 30 03:56:12.883184 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.883178    4528 scope.go:122] "RemoveContainer" containerID="05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"
Oct 30 03:56:12.883443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.883425    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0"} err="failed to get container status \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": rpc error: code = NotFound desc = could not find container \"05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0\": container with ID starting with 05dd31eb25da58c4f0186afa7ec950b08e55c248fdeff109cdffe94fdccd60d0 not found: ID does not exist"
Oct 30 03:56:12.883443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.883442    4528 scope.go:122] "RemoveContainer" containerID="4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"
Oct 30 03:56:12.883697 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.883673    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b"} err="failed to get container status \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": rpc error: code = NotFound desc = could not find container \"4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b\": container with ID starting with 4c57df780925ea7bef7c9e2a3eebc45581984e98a8c0a763b9d406838e94fe7b not found: ID does not exist"
Oct 30 03:56:12.912154 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.912129    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:12.912567 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.912532    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b0b6e223-8336-450b-b67b-27576e554cd3" containerName="otc-container" containerID="cri-o://d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72" gracePeriod=30
Oct 30 03:56:12.920304 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.920280    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:56:12.929572 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.929546    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 03:56:12.945119 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.945094    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:12.956173 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.956151    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:56:12.960458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:12.960435    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-pmv9m"]
Oct 30 03:56:13.073032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.072955    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:13.154554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.154530    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:13.189739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.189705    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:13.226427 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226394    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\") pod \"b0b6e223-8336-450b-b67b-27576e554cd3\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") "
Oct 30 03:56:13.226584 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226443    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\" (UniqueName: \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\") pod \"b0b6e223-8336-450b-b67b-27576e554cd3\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") "
Oct 30 03:56:13.226659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226583    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\") pod \"b0b6e223-8336-450b-b67b-27576e554cd3\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") "
Oct 30 03:56:13.226659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226618    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\") pod \"b0b6e223-8336-450b-b67b-27576e554cd3\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") "
Oct 30 03:56:13.226783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226716    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\") pod \"b0b6e223-8336-450b-b67b-27576e554cd3\" (UID: \"b0b6e223-8336-450b-b67b-27576e554cd3\") "
Oct 30 03:56:13.227009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.226978    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal" pod "b0b6e223-8336-450b-b67b-27576e554cd3" (UID: "b0b6e223-8336-450b-b67b-27576e554cd3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:13.228851 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.228790    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm" pod "b0b6e223-8336-450b-b67b-27576e554cd3" (UID: "b0b6e223-8336-450b-b67b-27576e554cd3"). InnerVolumeSpecName "kube-api-access-99sdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:13.228957 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.228906    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs" pod "b0b6e223-8336-450b-b67b-27576e554cd3" (UID: "b0b6e223-8336-450b-b67b-27576e554cd3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:13.229344 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.229316    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs" pod "b0b6e223-8336-450b-b67b-27576e554cd3" (UID: "b0b6e223-8336-450b-b67b-27576e554cd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:13.234189 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.234159    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert" pod "b0b6e223-8336-450b-b67b-27576e554cd3" (UID: "b0b6e223-8336-450b-b67b-27576e554cd3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:13.328042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.327972    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:13.328042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.328024    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:13.328042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.328034    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b6e223-8336-450b-b67b-27576e554cd3-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:13.328042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.328043    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99sdm\" (UniqueName: \"kubernetes.io/projected/b0b6e223-8336-450b-b67b-27576e554cd3-kube-api-access-99sdm\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:13.328303 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.328053    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0b6e223-8336-450b-b67b-27576e554cd3-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:56:13.813434 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.813399    4528 generic.go:356] "Generic (PLEG): container finished" podID="b0b6e223-8336-450b-b67b-27576e554cd3" containerID="d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72" exitCode=0
Oct 30 03:56:13.813932 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.813452    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:13.813932 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.813487    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b0b6e223-8336-450b-b67b-27576e554cd3","Type":"ContainerDied","Data":"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"}
Oct 30 03:56:13.813932 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.813554    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b0b6e223-8336-450b-b67b-27576e554cd3","Type":"ContainerDied","Data":"1c5fc55e9296f0974ecafa6096fb18022a3e64791db51ce11332d06c37ea2f8b"}
Oct 30 03:56:13.813932 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.813579    4528 scope.go:122] "RemoveContainer" containerID="d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"
Oct 30 03:56:13.822310 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.822291    4528 scope.go:122] "RemoveContainer" containerID="d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"
Oct 30 03:56:13.822549 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:13.822532    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72\": container with ID starting with d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72 not found: ID does not exist" containerID="d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"
Oct 30 03:56:13.822611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.822555    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72"} err="failed to get container status \"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72\": rpc error: code = NotFound desc = could not find container \"d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72\": container with ID starting with d2bef5938d3df5bae9c0e87723671ac61ffc02b09c47d5095950935094f5bc72 not found: ID does not exist"
Oct 30 03:56:13.857798 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.857769    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:13.861122 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.861097    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:13.891739 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.891706    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" path="/var/lib/kubelet/pods/5b687f0b-239a-4fad-95ee-51a0af2ea98a/volumes"
Oct 30 03:56:13.892242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.892221    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" path="/var/lib/kubelet/pods/6aeedf29-24be-4352-ac34-e1bb63cf8dc7/volumes"
Oct 30 03:56:13.892889 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.892813    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b6e223-8336-450b-b67b-27576e554cd3" path="/var/lib/kubelet/pods/b0b6e223-8336-450b-b67b-27576e554cd3/volumes"
Oct 30 03:56:13.893650 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.893626    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" path="/var/lib/kubelet/pods/ed237b85-bb8d-4c84-94a3-2a0ef65954f4/volumes"
Oct 30 03:56:13.963131 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:13.963102    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:14.122917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:14.122810    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:14.270258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:14.270222    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:14.744764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:14.744729    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:14.864784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:14.864735    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:18.095412 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.095378    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:18.211022 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.210983    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:18.326328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.326292    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:18.915965 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.915930    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:18.916680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916657    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="prom-label-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916683    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="prom-label-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916706    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="kube-rbac-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916714    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="kube-rbac-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="init-config-reloader"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916733    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="init-config-reloader"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916748    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="kube-rbac-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916755    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="kube-rbac-proxy"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916764    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="thanos-sidecar"
Oct 30 03:56:18.916774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916771    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="thanos-sidecar"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916782    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="config-reloader"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916789    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="config-reloader"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916803    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="prom-label-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916810    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="prom-label-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916834    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="prometheus"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916842    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="prometheus"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916864    4528 cpu_manager.go:414] "containerMap: removing container" podUID="29bd6358-a77c-4265-b6cf-3cd52eedb6ea" containerName="curl"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916872    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bd6358-a77c-4265-b6cf-3cd52eedb6ea" containerName="curl"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916884    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b0b6e223-8336-450b-b67b-27576e554cd3" containerName="otc-container"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916892    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b6e223-8336-450b-b67b-27576e554cd3" containerName="otc-container"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916960    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b6e223-8336-450b-b67b-27576e554cd3" containerName="otc-container"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916972    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="prom-label-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916982    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b687f0b-239a-4fad-95ee-51a0af2ea98a" containerName="kube-rbac-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.916992    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="kube-rbac-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.917000    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="thanos-sidecar"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.917009    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="config-reloader"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.917017    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aeedf29-24be-4352-ac34-e1bb63cf8dc7" containerName="prom-label-proxy"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.917024    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed237b85-bb8d-4c84-94a3-2a0ef65954f4" containerName="prometheus"
Oct 30 03:56:18.917109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.917035    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bd6358-a77c-4265-b6cf-3cd52eedb6ea" containerName="curl"
Oct 30 03:56:18.921777 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.921748    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:18.926924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.926897    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:18.946425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.946403    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:18.972927 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.972887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:18.973111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.972941    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:18.973111 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:18.972963    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jg4\" (UniqueName: \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.007414 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.007381    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:56:19.007625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.007568    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:56:19.007706 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.007628    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-ffv7w\""
Oct 30 03:56:19.007706 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.007695    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:56:19.007911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.007808    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 03:56:19.052523 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.052484    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:19.073623 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.073594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.073753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.073639    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.073753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.073662    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jg4\" (UniqueName: \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.074309 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.074289    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.076201 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.076178    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.095441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.095417    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jg4\" (UniqueName: \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\") pod \"data-science-collector-collector-1\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.161023 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.160973    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:19.237032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.236943    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:56:19.534548 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.534469    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:56:19.540172 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:56:19.540143    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec485878_0c4e_4442_baa1_dc4e5a6e7223.slice/crio-d4408c915cb63be08eabd17c2cbd9777a95d7526ba91c7a5ee0c154991f6c3ba WatchSource:0}: Error finding container d4408c915cb63be08eabd17c2cbd9777a95d7526ba91c7a5ee0c154991f6c3ba: Status 404 returned error can't find the container with id d4408c915cb63be08eabd17c2cbd9777a95d7526ba91c7a5ee0c154991f6c3ba
Oct 30 03:56:19.542388 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.542346    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:56:19.542468 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.542426    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:56:19.845612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.845518    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec485878-0c4e-4442-baa1-dc4e5a6e7223","Type":"ContainerStarted","Data":"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"}
Oct 30 03:56:19.845612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.845570    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec485878-0c4e-4442-baa1-dc4e5a6e7223","Type":"ContainerStarted","Data":"d4408c915cb63be08eabd17c2cbd9777a95d7526ba91c7a5ee0c154991f6c3ba"}
Oct 30 03:56:19.882330 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.882282    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.8822675009999998 podStartE2EDuration="1.882267501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:19.882136122 +0000 UTC m=+4538.106385399" watchObservedRunningTime="2026-10-30 03:56:19.882267501 +0000 UTC m=+4538.106516778"
Oct 30 03:56:19.947425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.947392    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:56:19.951225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.951199    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:19.964151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964125    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:56:19.964298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964152    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:56:19.964298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964169    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-xbzkc\""
Oct 30 03:56:19.964298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964213    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:56:19.964502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964485    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:56:19.964621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964603    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:56:19.964757 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964629    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:56:19.964757 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.964642    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:56:19.975847 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:19.975812    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:56:20.019949 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.019913    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:20.080987 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.080956    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.080999    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081088    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081262 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081163    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081262 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081189    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6d7\" (UniqueName: \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081262 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081210    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.081366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.081306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.137159 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.137120    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:20.182225 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182193    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182233    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182275    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182331    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6d7\" (UniqueName: \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.182726 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.182419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.183149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.183116    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.183149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.183144    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.183337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.183119    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.183337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.183193    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.184850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.184810    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.184850 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.184845    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.185009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.184987    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:20.203945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:20.203922    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6d7\" (UniqueName: \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:26.932514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:26.932427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:26.956871 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:26.956835    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.033314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.033280    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.033441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.033409    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.045759 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.045736    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.146480 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.146441    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.170383 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.170355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.468829 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.468787    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:27.794568 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.794475    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:56:27.798656 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:56:27.798627    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b576af4_f372_4eb0_bdf1_1724273abf6e.slice/crio-4507f35042bcd75a0c13b04de540e10a2f3e9f84266a14175cb0cd6b359a1c23 WatchSource:0}: Error finding container 4507f35042bcd75a0c13b04de540e10a2f3e9f84266a14175cb0cd6b359a1c23: Status 404 returned error can't find the container with id 4507f35042bcd75a0c13b04de540e10a2f3e9f84266a14175cb0cd6b359a1c23
Oct 30 03:56:27.881591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.881562    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerStarted","Data":"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"}
Oct 30 03:56:27.881712 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:27.881623    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerStarted","Data":"4507f35042bcd75a0c13b04de540e10a2f3e9f84266a14175cb0cd6b359a1c23"}
Oct 30 03:56:28.888069 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.888031    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerStarted","Data":"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"}
Oct 30 03:56:28.888475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.888082    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerStarted","Data":"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"}
Oct 30 03:56:28.888475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.888352    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:28.888475 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.888388    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:28.890806 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.890788    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:28.896441 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.896423    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:28.951231 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:28.951187    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.951175202 podStartE2EDuration="9.951175202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:28.940641099 +0000 UTC m=+4547.164890375" watchObservedRunningTime="2026-10-30 03:56:28.951175202 +0000 UTC m=+4547.175424475"
Oct 30 03:56:29.892901 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:29.892872    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:56:34.631433 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:56:34.631398    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:56:38.972207 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:38.972169    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:56:51.902617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:56:51.902574    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:57:01.901777 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:01.901718    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:57:04.630179 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:04.630135    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:57:11.902165 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:11.902134    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:14.389502 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.389443    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:14.390002 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.389677    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" containerName="otc-container" containerID="cri-o://355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50" gracePeriod=30
Oct 30 03:57:14.470361 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.470330    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:14.595338 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.595299    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:14.638509 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.638483    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:14.736658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.736580    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\" (UniqueName: \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\") pod \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") "
Oct 30 03:57:14.736658 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.736646    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\") pod \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") "
Oct 30 03:57:14.736865 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.736677    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\") pod \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\" (UID: \"ec485878-0c4e-4442-baa1-dc4e5a6e7223\") "
Oct 30 03:57:14.737117 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.737095    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal" pod "ec485878-0c4e-4442-baa1-dc4e5a6e7223" (UID: "ec485878-0c4e-4442-baa1-dc4e5a6e7223"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:14.738774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.738745    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4" pod "ec485878-0c4e-4442-baa1-dc4e5a6e7223" (UID: "ec485878-0c4e-4442-baa1-dc4e5a6e7223"). InnerVolumeSpecName "kube-api-access-85jg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:14.738774 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.738746    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs" pod "ec485878-0c4e-4442-baa1-dc4e5a6e7223" (UID: "ec485878-0c4e-4442-baa1-dc4e5a6e7223"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:14.784153 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.784120    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:14.837493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.837453    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec485878-0c4e-4442-baa1-dc4e5a6e7223-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:14.837493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.837487    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jg4\" (UniqueName: \"kubernetes.io/projected/ec485878-0c4e-4442-baa1-dc4e5a6e7223-kube-api-access-85jg4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:14.837686 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:14.837503    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec485878-0c4e-4442-baa1-dc4e5a6e7223-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:15.032186 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.032092    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:15.032470 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.032413    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway-opa" containerID="cri-o://7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f" gracePeriod=30
Oct 30 03:57:15.032641 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.032599    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway" containerID="cri-o://94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68" gracePeriod=30
Oct 30 03:57:15.032757 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.032713    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo" containerID="cri-o://37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e" gracePeriod=30
Oct 30 03:57:15.090917 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.090870    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:15.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.090862    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec485878-0c4e-4442-baa1-dc4e5a6e7223","Type":"ContainerDied","Data":"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"}
Oct 30 03:57:15.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.090950    4528 generic.go:356] "Generic (PLEG): container finished" podID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" containerID="355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50" exitCode=0
Oct 30 03:57:15.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.091027    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec485878-0c4e-4442-baa1-dc4e5a6e7223","Type":"ContainerDied","Data":"d4408c915cb63be08eabd17c2cbd9777a95d7526ba91c7a5ee0c154991f6c3ba"}
Oct 30 03:57:15.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.091055    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:15.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.091061    4528 scope.go:122] "RemoveContainer" containerID="355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"
Oct 30 03:57:15.136045 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.136015    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:15.140226 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.140202    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:15.143678 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.143658    4528 scope.go:122] "RemoveContainer" containerID="355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"
Oct 30 03:57:15.144122 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:15.144098    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50\": container with ID starting with 355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50 not found: ID does not exist" containerID="355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"
Oct 30 03:57:15.144229 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.144132    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50"} err="failed to get container status \"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50\": rpc error: code = NotFound desc = could not find container \"355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50\": container with ID starting with 355fdd4701361fefc14131e63b5554639a5301ad055f57fca169ff04c2d83e50 not found: ID does not exist"
Oct 30 03:57:15.215664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.215611    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:15.891721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:15.891690    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" path="/var/lib/kubelet/pods/ec485878-0c4e-4442-baa1-dc4e5a6e7223/volumes"
Oct 30 03:57:16.098514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.098485    4528 generic.go:356] "Generic (PLEG): container finished" podID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerID="7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f" exitCode=0
Oct 30 03:57:16.098514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.098508    4528 generic.go:356] "Generic (PLEG): container finished" podID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerID="37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e" exitCode=0
Oct 30 03:57:16.098729 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.098544    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerDied","Data":"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"}
Oct 30 03:57:16.098729 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.098583    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerDied","Data":"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"}
Oct 30 03:57:16.278366 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.278341    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:16.347176 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347145    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347176 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347180    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347392 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347213    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\" (UniqueName: \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347392 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347246    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347500 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347381    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347500 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347427    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347510    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347612 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347569    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347727 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347653    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"5b576af4-f372-4eb0-bdf1-1724273abf6e\" (UID: \"5b576af4-f372-4eb0-bdf1-1724273abf6e\") "
Oct 30 03:57:16.347727 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347675    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:16.347727 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347706    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:16.347911 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347790    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:16.347976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347940    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.347976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347960    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.348052 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.347976    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.348052 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.348004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:16.349789 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.349693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "kube-api-access-ht6d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:16.349945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.349801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:16.350105 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.350068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:16.350440 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.350412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:16.351108 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.351075    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "5b576af4-f372-4eb0-bdf1-1724273abf6e" (UID: "5b576af4-f372-4eb0-bdf1-1724273abf6e"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:57:16.449359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449276    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.449359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449327    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:57:16.449359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449345    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6d7\" (UniqueName: \"kubernetes.io/projected/5b576af4-f372-4eb0-bdf1-1724273abf6e-kube-api-access-ht6d7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.449554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449360    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.449554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449376    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.449554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.449390    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/5b576af4-f372-4eb0-bdf1-1724273abf6e-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:16.457442 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.457416    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:57:16.550006 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:16.549972    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:57:17.105257 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.105216    4528 generic.go:356] "Generic (PLEG): container finished" podID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerID="94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68" exitCode=0
Oct 30 03:57:17.105702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.105261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerDied","Data":"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"}
Oct 30 03:57:17.105702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.105278    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:17.105702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.105320    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"5b576af4-f372-4eb0-bdf1-1724273abf6e","Type":"ContainerDied","Data":"4507f35042bcd75a0c13b04de540e10a2f3e9f84266a14175cb0cd6b359a1c23"}
Oct 30 03:57:17.105702 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.105351    4528 scope.go:122] "RemoveContainer" containerID="7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"
Oct 30 03:57:17.114076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.114056    4528 scope.go:122] "RemoveContainer" containerID="94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"
Oct 30 03:57:17.122065 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.122042    4528 scope.go:122] "RemoveContainer" containerID="37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"
Oct 30 03:57:17.129380 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.129363    4528 scope.go:122] "RemoveContainer" containerID="7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"
Oct 30 03:57:17.129668 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:17.129637    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f\": container with ID starting with 7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f not found: ID does not exist" containerID="7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"
Oct 30 03:57:17.129717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.129676    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f"} err="failed to get container status \"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f\": rpc error: code = NotFound desc = could not find container \"7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f\": container with ID starting with 7c8d7e674999a86ca3121c0489a08642394cb512496223ded1be1dcb0360902f not found: ID does not exist"
Oct 30 03:57:17.129717 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.129695    4528 scope.go:122] "RemoveContainer" containerID="94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"
Oct 30 03:57:17.129954 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:17.129935    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68\": container with ID starting with 94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68 not found: ID does not exist" containerID="94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"
Oct 30 03:57:17.130067 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.129958    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68"} err="failed to get container status \"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68\": rpc error: code = NotFound desc = could not find container \"94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68\": container with ID starting with 94a8fe37828f2823aff9c201df5398a7a3ddfc684b7a6a7f16ef088e07b52d68 not found: ID does not exist"
Oct 30 03:57:17.130067 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.129976    4528 scope.go:122] "RemoveContainer" containerID="37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"
Oct 30 03:57:17.130242 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:17.130224    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e\": container with ID starting with 37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e not found: ID does not exist" containerID="37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"
Oct 30 03:57:17.130277 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.130247    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e"} err="failed to get container status \"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e\": rpc error: code = NotFound desc = could not find container \"37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e\": container with ID starting with 37edbef268410ac3b43e6e7fbe4673df50df4484346b40853dc944b93927280e not found: ID does not exist"
Oct 30 03:57:17.160614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.160577    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:17.170416 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.170379    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:17.305242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.305210    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:17.421451 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.421405    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:17.594445 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.594409    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:17.892707 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:17.892666    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" path="/var/lib/kubelet/pods/5b576af4-f372-4eb0-bdf1-1724273abf6e/volumes"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.015572    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.015979    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" containerName="otc-container"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.015996    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" containerName="otc-container"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016010    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway-opa"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016018    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway-opa"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016051    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016060    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016072    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway"
Oct 30 03:57:18.016076 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016080    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway"
Oct 30 03:57:18.016642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016156    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo"
Oct 30 03:57:18.016642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016169    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway"
Oct 30 03:57:18.016642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016181    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec485878-0c4e-4442-baa1-dc4e5a6e7223" containerName="otc-container"
Oct 30 03:57:18.016642 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.016189    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b576af4-f372-4eb0-bdf1-1724273abf6e" containerName="tempo-gateway-opa"
Oct 30 03:57:18.021032 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.021009    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.042394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.042363    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:18.068977 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.068941    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.114154 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.114126    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:57:18.114569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.114192    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:57:18.114569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.114262    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wlrhc\""
Oct 30 03:57:18.114569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.114305    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 03:57:18.114569 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.114332    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:57:18.165431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.165350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfz5\" (UniqueName: \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.165431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.165386    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.165603 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.165495    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.181190 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.181142    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.265860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.265795    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.266017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.265896    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfz5\" (UniqueName: \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.266017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.265924    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.266633 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.266609    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.268356 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.268334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.286792 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.286765    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfz5\" (UniqueName: \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\") pod \"data-science-collector-collector-1\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.299363 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.299340    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:18.336535 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.336506    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:18.630680 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.630641    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:18.635617 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:57:18.635583    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96e92c3_2f03_4a7b_b47f_a9437a377fc5.slice/crio-fe01c030d033d6b1b85f0a757219f73c730f9b3e35af4c11e981f56273ced9ca WatchSource:0}: Error finding container fe01c030d033d6b1b85f0a757219f73c730f9b3e35af4c11e981f56273ced9ca: Status 404 returned error can't find the container with id fe01c030d033d6b1b85f0a757219f73c730f9b3e35af4c11e981f56273ced9ca
Oct 30 03:57:18.637741 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.637711    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:57:18.637859 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:18.637772    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:57:19.050098 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.049997    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:19.054202 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.054177    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.068700 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068672    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:57:19.068700 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068674    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:57:19.068925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068676    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:57:19.068925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068715    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:57:19.068925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068765    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:57:19.068925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.068850    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-5ps4b\""
Oct 30 03:57:19.069109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.069004    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:57:19.069109 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.069070    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:57:19.080443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.080339    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:19.098813 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.098786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:19.116756 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.116729    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a96e92c3-2f03-4a7b-b47f-a9437a377fc5","Type":"ContainerStarted","Data":"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"}
Oct 30 03:57:19.117161 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.116774    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a96e92c3-2f03-4a7b-b47f-a9437a377fc5","Type":"ContainerStarted","Data":"fe01c030d033d6b1b85f0a757219f73c730f9b3e35af4c11e981f56273ced9ca"}
Oct 30 03:57:19.153786 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.153731    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.153718041 podStartE2EDuration="2.153718041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:19.15248949 +0000 UTC m=+4597.376738766" watchObservedRunningTime="2026-10-30 03:57:19.153718041 +0000 UTC m=+4597.377967325"
Oct 30 03:57:19.172918 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.172879    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173105 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173206 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173133    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173298    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173378 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173347    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173378 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173373    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptzg\" (UniqueName: \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173497 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173480    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.173561 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.173546    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.249737 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.249698    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:19.274369 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274342    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274414    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274445    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274467    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptzg\" (UniqueName: \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274525 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274530    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.274896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.274598    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.275298 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.275276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.275401 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.275272    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.275530 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.275508    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.275566 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.275513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.276976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.276957    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.277133 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.277009    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.277177 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.277150    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:19.295732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:19.295708    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptzg\" (UniqueName: \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.353093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.353053    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.377316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.377289    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.454143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.454103    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.454359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.454337    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.467053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.467014    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.567090 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.567055    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.591223 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.591196    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.669694 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.669665    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:28.991975 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:28.991887    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:28.996496 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:57:28.996466    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0dc90f_e85a_489e_bf87_dc933bd83358.slice/crio-be20d4020e3d9c14c4fe15549236eeaa26e2ea565fbaae72d9825b0cd8feb73d WatchSource:0}: Error finding container be20d4020e3d9c14c4fe15549236eeaa26e2ea565fbaae72d9825b0cd8feb73d: Status 404 returned error can't find the container with id be20d4020e3d9c14c4fe15549236eeaa26e2ea565fbaae72d9825b0cd8feb73d
Oct 30 03:57:29.158862 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:29.158799    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerStarted","Data":"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"}
Oct 30 03:57:29.159004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:29.158894    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerStarted","Data":"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"}
Oct 30 03:57:29.159004 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:29.158934    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerStarted","Data":"be20d4020e3d9c14c4fe15549236eeaa26e2ea565fbaae72d9825b0cd8feb73d"}
Oct 30 03:57:30.164621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.164584    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerStarted","Data":"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"}
Oct 30 03:57:30.165143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.164862    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:30.165143 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.164898    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:30.167576 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.167529    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:30.173730 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.173710    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:30.227313 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:30.227265    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=11.227253655 podStartE2EDuration="11.227253655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:30.226447416 +0000 UTC m=+4608.450696693" watchObservedRunningTime="2026-10-30 03:57:30.227253655 +0000 UTC m=+4608.451502931"
Oct 30 03:57:31.169234 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:31.169204    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:33.455678 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.455641    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:33.457186 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.455911    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" containerName="otc-container" containerID="cri-o://640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387" gracePeriod=30
Oct 30 03:57:33.533279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.533248    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:33.657314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.657267    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:33.700938 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.700915    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:33.772847 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.772709    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:33.800197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.800170    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\" (UniqueName: \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\") pod \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") "
Oct 30 03:57:33.800367 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.800264    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\") pod \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") "
Oct 30 03:57:33.800367 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.800294    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\") pod \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\" (UID: \"a96e92c3-2f03-4a7b-b47f-a9437a377fc5\") "
Oct 30 03:57:33.800617 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.800591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal" pod "a96e92c3-2f03-4a7b-b47f-a9437a377fc5" (UID: "a96e92c3-2f03-4a7b-b47f-a9437a377fc5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:33.802172 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.802151    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5" pod "a96e92c3-2f03-4a7b-b47f-a9437a377fc5" (UID: "a96e92c3-2f03-4a7b-b47f-a9437a377fc5"). InnerVolumeSpecName "kube-api-access-6tfz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:33.802262 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.802241    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs" pod "a96e92c3-2f03-4a7b-b47f-a9437a377fc5" (UID: "a96e92c3-2f03-4a7b-b47f-a9437a377fc5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:33.901168 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.901141    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfz5\" (UniqueName: \"kubernetes.io/projected/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-kube-api-access-6tfz5\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:33.901168 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.901168    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:33.901331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.901177    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96e92c3-2f03-4a7b-b47f-a9437a377fc5-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:33.943404 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.943362    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:33.943696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.943611    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo" containerID="cri-o://d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d" gracePeriod=30
Oct 30 03:57:33.943696 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.943659    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway-opa" containerID="cri-o://165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4" gracePeriod=30
Oct 30 03:57:33.943919 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:33.943658    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway" containerID="cri-o://200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312" gracePeriod=30
Oct 30 03:57:34.013359 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.013323    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:34.122876 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.122812    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:34.184933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.184907    4528 generic.go:356] "Generic (PLEG): container finished" podID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerID="165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4" exitCode=0
Oct 30 03:57:34.184933 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.184928    4528 generic.go:356] "Generic (PLEG): container finished" podID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerID="d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d" exitCode=0
Oct 30 03:57:34.185133 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.184979    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerDied","Data":"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"}
Oct 30 03:57:34.185133 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.185031    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerDied","Data":"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"}
Oct 30 03:57:34.186204 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.186184    4528 generic.go:356] "Generic (PLEG): container finished" podID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" containerID="640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387" exitCode=0
Oct 30 03:57:34.186314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.186231    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:34.186314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.186227    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a96e92c3-2f03-4a7b-b47f-a9437a377fc5","Type":"ContainerDied","Data":"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"}
Oct 30 03:57:34.186393 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.186343    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a96e92c3-2f03-4a7b-b47f-a9437a377fc5","Type":"ContainerDied","Data":"fe01c030d033d6b1b85f0a757219f73c730f9b3e35af4c11e981f56273ced9ca"}
Oct 30 03:57:34.186393 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.186365    4528 scope.go:122] "RemoveContainer" containerID="640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"
Oct 30 03:57:34.193667 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.193650    4528 scope.go:122] "RemoveContainer" containerID="640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"
Oct 30 03:57:34.193928 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:34.193911    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387\": container with ID starting with 640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387 not found: ID does not exist" containerID="640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"
Oct 30 03:57:34.193993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.193943    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387"} err="failed to get container status \"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387\": rpc error: code = NotFound desc = could not find container \"640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387\": container with ID starting with 640bab42374ee86a2fc7e510257d3ae53d346dba888cc8eccea1ab6161773387 not found: ID does not exist"
Oct 30 03:57:34.229342 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.229318    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:34.233777 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:34.233758    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:34.630479 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:34.630448    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:57:35.186460 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.186437    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:35.192221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.192197    4528 generic.go:356] "Generic (PLEG): container finished" podID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerID="200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312" exitCode=0
Oct 30 03:57:35.192336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.192245    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:35.192336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.192245    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerDied","Data":"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"}
Oct 30 03:57:35.192336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.192299    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1f0dc90f-e85a-489e-bf87-dc933bd83358","Type":"ContainerDied","Data":"be20d4020e3d9c14c4fe15549236eeaa26e2ea565fbaae72d9825b0cd8feb73d"}
Oct 30 03:57:35.192336 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.192324    4528 scope.go:122] "RemoveContainer" containerID="165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"
Oct 30 03:57:35.199625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.199594    4528 scope.go:122] "RemoveContainer" containerID="200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"
Oct 30 03:57:35.206420 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.206403    4528 scope.go:122] "RemoveContainer" containerID="d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"
Oct 30 03:57:35.213194 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213177    4528 scope.go:122] "RemoveContainer" containerID="165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"
Oct 30 03:57:35.213449 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:35.213432    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4\": container with ID starting with 165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4 not found: ID does not exist" containerID="165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"
Oct 30 03:57:35.213493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213457    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4"} err="failed to get container status \"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4\": rpc error: code = NotFound desc = could not find container \"165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4\": container with ID starting with 165499dc8801a9da7f3721914e693628c97deb648a7f10b08346e5cc8e41b3a4 not found: ID does not exist"
Oct 30 03:57:35.213493 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213474    4528 scope.go:122] "RemoveContainer" containerID="200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"
Oct 30 03:57:35.213693 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:35.213677    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312\": container with ID starting with 200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312 not found: ID does not exist" containerID="200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"
Oct 30 03:57:35.213735 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213698    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312"} err="failed to get container status \"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312\": rpc error: code = NotFound desc = could not find container \"200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312\": container with ID starting with 200ea66dadabcfbcd1e04e541ff454bb78e59b5694c5237e8dbe532a33740312 not found: ID does not exist"
Oct 30 03:57:35.213735 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213711    4528 scope.go:122] "RemoveContainer" containerID="d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"
Oct 30 03:57:35.213938 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:57:35.213923    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d\": container with ID starting with d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d not found: ID does not exist" containerID="d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"
Oct 30 03:57:35.213976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.213941    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d"} err="failed to get container status \"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d\": rpc error: code = NotFound desc = could not find container \"d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d\": container with ID starting with d9010fc83f55b4ef988dfa9a0c6a815d1e134838eaec79ffb4745d8fe63b9e5d not found: ID does not exist"
Oct 30 03:57:35.312033 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.311953    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\" (UniqueName: \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312195 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312079    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312195 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312143    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312195 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312185    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312277    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312312    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312372 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.312591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312423    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"1f0dc90f-e85a-489e-bf87-dc933bd83358\" (UID: \"1f0dc90f-e85a-489e-bf87-dc933bd83358\") "
Oct 30 03:57:35.313009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.312869    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:35.313221 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.313115    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:35.313361 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.313250    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:35.313438 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.313418    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:35.314435 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.314411    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:35.314557 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.314492    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "kube-api-access-7ptzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:35.314625 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.314592    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:35.315020 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.314996    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:57:35.315305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.315287    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "1f0dc90f-e85a-489e-bf87-dc933bd83358" (UID: "1f0dc90f-e85a-489e-bf87-dc933bd83358"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:57:35.413331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413299    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413326    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413331 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413336    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptzg\" (UniqueName: \"kubernetes.io/projected/1f0dc90f-e85a-489e-bf87-dc933bd83358-kube-api-access-7ptzg\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413365    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413376    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413386    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413394    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413404    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.413554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.413412    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1f0dc90f-e85a-489e-bf87-dc933bd83358-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:57:35.422305 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.422287    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:57:35.514039 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.514012    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:57:35.531775 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.531748    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:35.541514 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.541491    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:35.892183 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.892152    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" path="/var/lib/kubelet/pods/1f0dc90f-e85a-489e-bf87-dc933bd83358/volumes"
Oct 30 03:57:35.892623 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:35.892610    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" path="/var/lib/kubelet/pods/a96e92c3-2f03-4a7b-b47f-a9437a377fc5/volumes"
Oct 30 03:57:36.159771 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.159689    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:36.165059 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.165027    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.70:8083/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:57:36.165138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.165032    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.133.0.70:8081/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:57:36.279872 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.279834    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:36.387730 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.387698    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:36.507399 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.507315    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:36.893007 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.892976    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893249    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway-opa"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893259    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway-opa"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893280    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" containerName="otc-container"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893285    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" containerName="otc-container"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893296    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893304    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893311    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893318    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893366    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway-opa"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893380    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96e92c3-2f03-4a7b-b47f-a9437a377fc5" containerName="otc-container"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893388    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo"
Oct 30 03:57:36.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.893393    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0dc90f-e85a-489e-bf87-dc933bd83358" containerName="tempo-gateway"
Oct 30 03:57:36.898098 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.898074    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:36.919965 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.919925    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:36.924063 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.924041    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:36.941968 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.941936    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-frlgf\""
Oct 30 03:57:36.946160 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.946140    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:57:36.989492 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.989463    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:57:36.989492 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.989463    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:57:36.989492 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:36.989463    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 03:57:37.025804 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.025778    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.025963 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.025811    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.025963 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.025938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmgx\" (UniqueName: \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.046836 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.046788    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:37.126731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.126696    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.126924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.126738    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmgx\" (UniqueName: \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.126924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.126808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.127557 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.127535    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.129272 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.129248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.147732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.147668    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmgx\" (UniqueName: \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\") pod \"data-science-collector-collector-1\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.165031 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.164999    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:37.210801 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.210772    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:57:37.503582 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.503491    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:57:37.508471 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:57:37.508446    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a76a25_42f9_4fc8_babf_4af15e06432d.slice/crio-6f11eaba7cd95b4f1342d034c96b78a4cb49acd7d33409187b18e37b2e148a24 WatchSource:0}: Error finding container 6f11eaba7cd95b4f1342d034c96b78a4cb49acd7d33409187b18e37b2e148a24: Status 404 returned error can't find the container with id 6f11eaba7cd95b4f1342d034c96b78a4cb49acd7d33409187b18e37b2e148a24
Oct 30 03:57:37.511074 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.511041    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:57:37.511168 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.511097    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:57:37.729897 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.729799    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:37.733660 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.733637    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.746138 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.746094    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 03:57:37.746753 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.746731    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747082    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747128    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747135    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-f2s7v\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747145    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747160    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:57:37.747197 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.747128    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 03:57:37.757064 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.757012    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:37.787784 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.787751    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:37.834081 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834092    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834121    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834273    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834384    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.834545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.834416    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2rx\" (UniqueName: \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.896236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.896201    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:37.935311 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935285    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935313    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935383    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2rx\" (UniqueName: \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935449 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935428    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935633 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935464    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.935633 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.935492    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.936141 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.936099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.936264 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.936108    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.936342 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.936296    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.936481 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.936456    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.938302 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.938274    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.938443 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.938419    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.938491 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.938423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:37.956381 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:37.956360    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2rx\" (UniqueName: \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.211342 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.211296    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d1a76a25-42f9-4fc8-babf-4af15e06432d","Type":"ContainerStarted","Data":"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"}
Oct 30 03:57:38.211545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.211354    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d1a76a25-42f9-4fc8-babf-4af15e06432d","Type":"ContainerStarted","Data":"6f11eaba7cd95b4f1342d034c96b78a4cb49acd7d33409187b18e37b2e148a24"}
Oct 30 03:57:38.248638 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.248590    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.248577897 podStartE2EDuration="2.248577897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:38.247481314 +0000 UTC m=+4616.471730591" watchObservedRunningTime="2026-10-30 03:57:38.248577897 +0000 UTC m=+4616.472827174"
Oct 30 03:57:38.641210 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.641175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.665228 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.665194    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.742376 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.742338    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.742542 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.742474    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.755053 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.755029    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.851731 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.851695    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.875732 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.875703    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:38.949269 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:38.949172    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:39.110722 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:39.110685    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:57:39.275307 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:39.275178    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:57:39.280505 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:57:39.280479    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d063d8b_16f2_4bac_9f4d_90fa1aa37a42.slice/crio-31b5fd173fd9fcf7668fddf53e9db49b9df70e4dbaf2c6964a52bb41d74cfdb1 WatchSource:0}: Error finding container 31b5fd173fd9fcf7668fddf53e9db49b9df70e4dbaf2c6964a52bb41d74cfdb1: Status 404 returned error can't find the container with id 31b5fd173fd9fcf7668fddf53e9db49b9df70e4dbaf2c6964a52bb41d74cfdb1
Oct 30 03:57:40.224123 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224084    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerStarted","Data":"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"}
Oct 30 03:57:40.224611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224159    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:40.224611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224187    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerStarted","Data":"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"}
Oct 30 03:57:40.224611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224330    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:40.224611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerStarted","Data":"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"}
Oct 30 03:57:40.224611 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.224435    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerStarted","Data":"31b5fd173fd9fcf7668fddf53e9db49b9df70e4dbaf2c6964a52bb41d74cfdb1"}
Oct 30 03:57:40.226643 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.226622    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:40.232486 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.232470    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:40.285128 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:40.285082    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.285070191 podStartE2EDuration="3.285070191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:40.275053976 +0000 UTC m=+4618.499303253" watchObservedRunningTime="2026-10-30 03:57:40.285070191 +0000 UTC m=+4618.509319468"
Oct 30 03:57:41.228087 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:41.228055    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:57:42.451627 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:42.451597    4528 scope.go:122] "RemoveContainer" containerID="7da3d557559e9bf650f30f239edd2691fffca7fbaf79f09d586a6a252145f915"
Oct 30 03:57:42.459430 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:57:42.459409    4528 scope.go:122] "RemoveContainer" containerID="d8037d242b997cee58212a6d1ff27047ee657c6eb2511263c3f57171a41abd24"
Oct 30 03:58:02.234253 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:02.234145    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:58:04.630665 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:04.630630    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:58:12.234030 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:12.233990    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:58:22.233374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:22.233330    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:58:24.405904 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.405868    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:24.406316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.406062    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d1a76a25-42f9-4fc8-babf-4af15e06432d" containerName="otc-container" containerID="cri-o://5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7" gracePeriod=30
Oct 30 03:58:24.470764 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.470722    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:24.654483 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.654460    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:24.668242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.668165    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:24.682050 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.681751    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\" (UniqueName: \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\") pod \"d1a76a25-42f9-4fc8-babf-4af15e06432d\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") "
Oct 30 03:58:24.682050 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.681833    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\") pod \"d1a76a25-42f9-4fc8-babf-4af15e06432d\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") "
Oct 30 03:58:24.682289 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.682264    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal" pod "d1a76a25-42f9-4fc8-babf-4af15e06432d" (UID: "d1a76a25-42f9-4fc8-babf-4af15e06432d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:24.684299 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.684266    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx" pod "d1a76a25-42f9-4fc8-babf-4af15e06432d" (UID: "d1a76a25-42f9-4fc8-babf-4af15e06432d"). InnerVolumeSpecName "kube-api-access-xvmgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:24.782527 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.782498    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\") pod \"d1a76a25-42f9-4fc8-babf-4af15e06432d\" (UID: \"d1a76a25-42f9-4fc8-babf-4af15e06432d\") "
Oct 30 03:58:24.782674 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.782629    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvmgx\" (UniqueName: \"kubernetes.io/projected/d1a76a25-42f9-4fc8-babf-4af15e06432d-kube-api-access-xvmgx\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:24.782674 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.782643    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d1a76a25-42f9-4fc8-babf-4af15e06432d-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:24.784554 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.784520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs" pod "d1a76a25-42f9-4fc8-babf-4af15e06432d" (UID: "d1a76a25-42f9-4fc8-babf-4af15e06432d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:24.806540 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.806516    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:24.883142 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.883102    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a76a25-42f9-4fc8-babf-4af15e06432d-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:24.972252 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.972165    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:58:24.972510 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.972477    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo" containerID="cri-o://f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456" gracePeriod=30
Oct 30 03:58:24.972689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.972627    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway-opa" containerID="cri-o://a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b" gracePeriod=30
Oct 30 03:58:24.972689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:24.972646    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway" containerID="cri-o://bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa" gracePeriod=30
Oct 30 03:58:25.045478 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.045443    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:25.183247 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.183214    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:25.225429 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.225350    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.72:8083/ready\": dial tcp 10.133.0.72:8083: connect: connection refused"
Oct 30 03:58:25.229150 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.229124    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:58:25.406464 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.406428    4528 generic.go:356] "Generic (PLEG): container finished" podID="d1a76a25-42f9-4fc8-babf-4af15e06432d" containerID="5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7" exitCode=0
Oct 30 03:58:25.406941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.406469    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:25.406941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.406519    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d1a76a25-42f9-4fc8-babf-4af15e06432d","Type":"ContainerDied","Data":"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"}
Oct 30 03:58:25.406941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.406569    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d1a76a25-42f9-4fc8-babf-4af15e06432d","Type":"ContainerDied","Data":"6f11eaba7cd95b4f1342d034c96b78a4cb49acd7d33409187b18e37b2e148a24"}
Oct 30 03:58:25.406941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.406601    4528 scope.go:122] "RemoveContainer" containerID="5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"
Oct 30 03:58:25.408895 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.408873    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerID="a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b" exitCode=0
Oct 30 03:58:25.408895 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.408893    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerID="f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456" exitCode=0
Oct 30 03:58:25.409029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.408947    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerDied","Data":"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"}
Oct 30 03:58:25.409029 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.409006    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerDied","Data":"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"}
Oct 30 03:58:25.416037 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.416020    4528 scope.go:122] "RemoveContainer" containerID="5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"
Oct 30 03:58:25.416283 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:25.416263    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7\": container with ID starting with 5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7 not found: ID does not exist" containerID="5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"
Oct 30 03:58:25.416328 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.416290    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7"} err="failed to get container status \"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7\": rpc error: code = NotFound desc = could not find container \"5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7\": container with ID starting with 5f294a8bf51816eef4946c11b5f082f5701afd67eb9b048fc0c95571271521a7 not found: ID does not exist"
Oct 30 03:58:25.452854 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.452810    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:25.456553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.456528    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:25.892587 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:25.892558    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a76a25-42f9-4fc8-babf-4af15e06432d" path="/var/lib/kubelet/pods/d1a76a25-42f9-4fc8-babf-4af15e06432d/volumes"
Oct 30 03:58:26.218560 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.218535    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:58:26.291812 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.291773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.291993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.291850    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.291993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.291894    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\" (UniqueName: \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.291993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.291926    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.291993 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.291962    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.292239 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292092    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.292239 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.292239 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292174    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.292239 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292202    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\") pod \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\" (UID: \"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42\") "
Oct 30 03:58:26.292454 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292259    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:26.292568 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:26.292671 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292572    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.292976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292936    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:26.293100 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.292969    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:26.294555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.294518    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:26.294834 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.294771    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:26.295047 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.295017    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "kube-api-access-bl2rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:26.295113 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.295065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:26.295501 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.295481    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" (UID: "6d063d8b-16f2-4bac-9f4d-90fa1aa37a42"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 03:58:26.393705 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393673    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393705 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393701    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2rx\" (UniqueName: \"kubernetes.io/projected/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-kube-api-access-bl2rx\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393705 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393712    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393723    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393750    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 03:58:26.393976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393761    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393770    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.393976 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.393795    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:26.402156 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.402138    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 03:58:26.415486 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.415459    4528 generic.go:356] "Generic (PLEG): container finished" podID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerID="bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa" exitCode=0
Oct 30 03:58:26.415896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.415512    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 03:58:26.415896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.415534    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerDied","Data":"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"}
Oct 30 03:58:26.415896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.415628    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"6d063d8b-16f2-4bac-9f4d-90fa1aa37a42","Type":"ContainerDied","Data":"31b5fd173fd9fcf7668fddf53e9db49b9df70e4dbaf2c6964a52bb41d74cfdb1"}
Oct 30 03:58:26.415896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.415663    4528 scope.go:122] "RemoveContainer" containerID="a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"
Oct 30 03:58:26.428614 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.428592    4528 scope.go:122] "RemoveContainer" containerID="bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"
Oct 30 03:58:26.435793 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.435776    4528 scope.go:122] "RemoveContainer" containerID="f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"
Oct 30 03:58:26.443072 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443056    4528 scope.go:122] "RemoveContainer" containerID="a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"
Oct 30 03:58:26.443362 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:26.443343    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b\": container with ID starting with a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b not found: ID does not exist" containerID="a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"
Oct 30 03:58:26.443420 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443369    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b"} err="failed to get container status \"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b\": rpc error: code = NotFound desc = could not find container \"a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b\": container with ID starting with a92fff7f9af5d0ae910aca85f8bfe25635e2a2d66706d098ab7ce6d40b16839b not found: ID does not exist"
Oct 30 03:58:26.443420 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443388    4528 scope.go:122] "RemoveContainer" containerID="bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"
Oct 30 03:58:26.443628 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:26.443609    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa\": container with ID starting with bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa not found: ID does not exist" containerID="bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"
Oct 30 03:58:26.443674 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443635    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa"} err="failed to get container status \"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa\": rpc error: code = NotFound desc = could not find container \"bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa\": container with ID starting with bda6591b23a123cc182003170d2334bfb00f5d956fd786fb48e81d89a19c7efa not found: ID does not exist"
Oct 30 03:58:26.443674 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443650    4528 scope.go:122] "RemoveContainer" containerID="f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"
Oct 30 03:58:26.443900 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:26.443879    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456\": container with ID starting with f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456 not found: ID does not exist" containerID="f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"
Oct 30 03:58:26.443963 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.443908    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456"} err="failed to get container status \"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456\": rpc error: code = NotFound desc = could not find container \"f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456\": container with ID starting with f0600c31706a0e41a6e247f8c183c9470d75434ccb77657edd20df24c110d456 not found: ID does not exist"
Oct 30 03:58:26.459241 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.459183    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:58:26.463830 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.463796    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 03:58:26.494216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:26.494186    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 03:58:27.892309 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:27.892272    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" path="/var/lib/kubelet/pods/6d063d8b-16f2-4bac-9f4d-90fa1aa37a42/volumes"
Oct 30 03:58:28.137230 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:28.137188    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:28.255980 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:28.255890    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:34.631385 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:58:34.631346    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:58:39.222432 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:39.222398    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:46.351969 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.351933    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"]
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352199    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d1a76a25-42f9-4fc8-babf-4af15e06432d" containerName="otc-container"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352210    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a76a25-42f9-4fc8-babf-4af15e06432d" containerName="otc-container"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352222    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352237    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352257    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352267    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway-opa"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352272    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway-opa"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352321    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352329    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352336    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d063d8b-16f2-4bac-9f4d-90fa1aa37a42" containerName="tempo-gateway-opa"
Oct 30 03:58:46.352373 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.352344    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a76a25-42f9-4fc8-babf-4af15e06432d" containerName="otc-container"
Oct 30 03:58:46.356943 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.356921    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:46.369699 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.369676    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:58:46.369814 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.369780    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 03:58:46.379281 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.379255    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:58:46.380308 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.380286    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"]
Oct 30 03:58:46.439448 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.439411    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfjms\" (UniqueName: \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\") pod \"seaweedfs-bucket-creator\" (UID: \"33452438-7732-4080-b8e4-77295c645cb1\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:46.540293 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.540251    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfjms\" (UniqueName: \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\") pod \"seaweedfs-bucket-creator\" (UID: \"33452438-7732-4080-b8e4-77295c645cb1\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:46.561425 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.561401    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfjms\" (UniqueName: \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\") pod \"seaweedfs-bucket-creator\" (UID: \"33452438-7732-4080-b8e4-77295c645cb1\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:46.667855 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.667811    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:46.959608 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.959532    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"]
Oct 30 03:58:46.964314 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:58:46.964287    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33452438_7732_4080_b8e4_77295c645cb1.slice/crio-9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2 WatchSource:0}: Error finding container 9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2: Status 404 returned error can't find the container with id 9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2
Oct 30 03:58:46.966105 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:46.966087    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:58:47.501954 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:47.501883    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"33452438-7732-4080-b8e4-77295c645cb1","Type":"ContainerStarted","Data":"9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2"}
Oct 30 03:58:52.526180 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:52.526145    4528 generic.go:356] "Generic (PLEG): container finished" podID="33452438-7732-4080-b8e4-77295c645cb1" containerID="1339c97d686204519f3df975f512177e8e75040dd30503ebd1b1e7d5071fbb2f" exitCode=0
Oct 30 03:58:52.526567 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:52.526202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"33452438-7732-4080-b8e4-77295c645cb1","Type":"ContainerDied","Data":"1339c97d686204519f3df975f512177e8e75040dd30503ebd1b1e7d5071fbb2f"}
Oct 30 03:58:53.659271 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:53.659248    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:53.702638 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:53.702610    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\" (UniqueName: \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\") pod \"33452438-7732-4080-b8e4-77295c645cb1\" (UID: \"33452438-7732-4080-b8e4-77295c645cb1\") "
Oct 30 03:58:53.704654 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:53.704626    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms" pod "33452438-7732-4080-b8e4-77295c645cb1" (UID: "33452438-7732-4080-b8e4-77295c645cb1"). InnerVolumeSpecName "kube-api-access-kfjms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:53.803749 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:53.803665    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfjms\" (UniqueName: \"kubernetes.io/projected/33452438-7732-4080-b8e4-77295c645cb1-kube-api-access-kfjms\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:58:54.535679 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:54.535638    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"33452438-7732-4080-b8e4-77295c645cb1","Type":"ContainerDied","Data":"9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2"}
Oct 30 03:58:54.535679 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:54.535667    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator"
Oct 30 03:58:54.535901 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:54.535688    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a81324a5a92e1372a034135b061614b095af3cc505bc305fe5cbd09a09ff5f2"
Oct 30 03:58:55.687017 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:55.686965    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:55.883329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:55.883289    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:56.031534 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.031459    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:56.380610 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.380572    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:56.380947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.380929    4528 cpu_manager.go:414] "containerMap: removing container" podUID="33452438-7732-4080-b8e4-77295c645cb1" containerName="create-bucket"
Oct 30 03:58:56.381024 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.380948    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33452438-7732-4080-b8e4-77295c645cb1" containerName="create-bucket"
Oct 30 03:58:56.381151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.381024    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="33452438-7732-4080-b8e4-77295c645cb1" containerName="create-bucket"
Oct 30 03:58:56.384042 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.384021    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.408040 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.408006    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:56.426838 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.426800    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.426998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.426910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.426998 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.426949    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztnl\" (UniqueName: \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.440872 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.440839    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:56.447755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.447721    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8l25s\""
Oct 30 03:58:56.447755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.447721    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 03:58:56.474961 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.474935    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 03:58:56.474961 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.474951    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\""
Oct 30 03:58:56.475125 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.474935    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 03:58:56.527918 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.527891    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.528054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.527926    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.528054 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.527942    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztnl\" (UniqueName: \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.528690 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.528670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.530279 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.530255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.549019 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.548996    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztnl\" (UniqueName: \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\") pod \"data-science-collector-collector-1\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.568637 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.568613    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:56.697469 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.697392    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 03:58:56.991322 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.991227    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 03:58:56.996792 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:58:56.996760    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f334f_1724_41dd_8081_08be2240cc79.slice/crio-f4ee5fa15c44ceee6dfcf210d999e09284ed9f9eb388aea25a18092cc91958b9 WatchSource:0}: Error finding container f4ee5fa15c44ceee6dfcf210d999e09284ed9f9eb388aea25a18092cc91958b9: Status 404 returned error can't find the container with id f4ee5fa15c44ceee6dfcf210d999e09284ed9f9eb388aea25a18092cc91958b9
Oct 30 03:58:56.998856 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.998792    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:58:56.998939 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:56.998900    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 03:58:57.551650 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:57.551622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4a1f334f-1724-41dd-8081-08be2240cc79","Type":"ContainerStarted","Data":"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"}
Oct 30 03:58:57.551794 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:57.551674    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4a1f334f-1724-41dd-8081-08be2240cc79","Type":"ContainerStarted","Data":"f4ee5fa15c44ceee6dfcf210d999e09284ed9f9eb388aea25a18092cc91958b9"}
Oct 30 03:58:57.588646 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:57.588596    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.588584532 podStartE2EDuration="1.588584532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:57.587595945 +0000 UTC m=+4695.811845219" watchObservedRunningTime="2026-10-30 03:58:57.588584532 +0000 UTC m=+4695.812833808"
Oct 30 03:58:59.480961 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.480922    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 03:58:59.484496 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.484473    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.501624 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.501599    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 03:58:59.511223 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.511192    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-zhr2x\""
Oct 30 03:58:59.511390 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.511305    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\""
Oct 30 03:58:59.511390 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.511352    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\""
Oct 30 03:58:59.512812 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.512791    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 03:58:59.541953 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.541922    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:59.554926 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.554895    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.555097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.554936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.555097 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.555057    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstrg\" (UniqueName: \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.555222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.555097    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.555222 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.555131    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.655624 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.655591    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstrg\" (UniqueName: \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.655800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.655635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.655800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.655667    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.655800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.655694    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.655800 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.655711    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.656185 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.656163    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.656357 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.656336    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.656555 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.656538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.658180 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.658159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.662945 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.662919    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:59.679721 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.679698    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstrg\" (UniqueName: \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\") pod \"tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.682952 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.682931    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 03:58:59.686938 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.686909    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.700860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.700838    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\""
Oct 30 03:58:59.700968 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.700954    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\""
Oct 30 03:58:59.708364 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.708338    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 03:58:59.728100 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.728067    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 03:58:59.731891 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.731808    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.754924 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.754885    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 03:58:59.757003 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.756980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.757142 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.757075    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.757211 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.757165    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.757270 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.757220    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjbmk\" (UniqueName: \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.757319 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.757279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.776621 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.776575    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:59.794940 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.794914    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 03:58:59.796836 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.796800    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:58:59.798545 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.798524    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.801374 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.801346    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\""
Oct 30 03:58:59.814337 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.814314    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\""
Oct 30 03:58:59.824093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.824066    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 03:58:59.859253 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.859220    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.859281    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.859394 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.859346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjbmk\" (UniqueName: \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.860453 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860420    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.860591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860519    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.860591 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860575    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.860716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860615    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.860716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860667    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs262\" (UniqueName: \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.860716 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860698    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860740    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nlb\" (UniqueName: \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860777    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860837    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860874    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860900    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860930    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.861028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.860957    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.861908 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.861882    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.862326 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.862304    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.867329 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.864809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.881431 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.881404    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjbmk\" (UniqueName: \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\") pod \"tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:58:59.889055 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.889009    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:58:59.961594 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961560    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs262\" (UniqueName: \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961632    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nlb\" (UniqueName: \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961703    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961736    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961766    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.961802 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.961809    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.962530 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962126    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.962530 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.962714 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962692    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.962783 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962762    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.962851 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962793    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.962893 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.962848    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.964562 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.964538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.964650 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.964542    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.982856 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.982772    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nlb\" (UniqueName: \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\") pod \"tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:58:59.982940 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.982854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs262\" (UniqueName: \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\") pod \"tempo-data-science-tempostack-compactor-589b765f47-nft5l\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:58:59.998635 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:58:59.998611    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:59:00.003914 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.003885    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:00.044710 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.044673    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:59:00.117983 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.117919    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 03:59:00.124384 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:59:00.124354    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b7ebc4_45da_47c3_88cb_c8647ed8a154.slice/crio-5539522619c70349f542144a77bae3457efa1e8738835ed1febbc27f27c0c950 WatchSource:0}: Error finding container 5539522619c70349f542144a77bae3457efa1e8738835ed1febbc27f27c0c950: Status 404 returned error can't find the container with id 5539522619c70349f542144a77bae3457efa1e8738835ed1febbc27f27c0c950
Oct 30 03:59:00.125848 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.125616    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:59:00.127974 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.127875    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:00.133094 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.133005    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:00.138193 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.138158    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.152886 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.152859    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 03:59:00.152886 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.152859    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:59:00.153912 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.153883    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 03:59:00.154028 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.154011    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-25rxl\""
Oct 30 03:59:00.154072 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.154059    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 03:59:00.154124 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.154113    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 03:59:00.162755 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.162726    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:00.264567 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264493    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264567 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264537    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh62t\" (UniqueName: \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264638    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264664    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264724 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264682    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.264879 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.264798    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.296153 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.296109    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 03:59:00.344787 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.344742    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 03:59:00.349385 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:59:00.349356    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde67c70c_bd18_4242_bed5_ab0ccecaa4fb.slice/crio-5bf62cbfce5f13d6f1dcfe2573d57aa758bb3869cc8def9a1e6c39710558baf4 WatchSource:0}: Error finding container 5bf62cbfce5f13d6f1dcfe2573d57aa758bb3869cc8def9a1e6c39710558baf4: Status 404 returned error can't find the container with id 5bf62cbfce5f13d6f1dcfe2573d57aa758bb3869cc8def9a1e6c39710558baf4
Oct 30 03:59:00.365376 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365386    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365406    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365495 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365451    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365527    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365561    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.365664 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.365588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh62t\" (UniqueName: \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.366249 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.366210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.366345 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.366223    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.366345 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.366270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.368008 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.367983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.368327 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.368309    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.368413 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.368396    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.386988 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.386965    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh62t\" (UniqueName: \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\") pod \"tempo-data-science-tempostack-gateway-665876f97-dxzvr\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.435941 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.435800    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 03:59:00.443890 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:59:00.443863    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9341140b_f736_4637_84c5_9deea3281a85.slice/crio-ebcc497de0385cec2534cd64d952b4ad03cd22e6b4fbdd9d19e5f71cbe861cb0 WatchSource:0}: Error finding container ebcc497de0385cec2534cd64d952b4ad03cd22e6b4fbdd9d19e5f71cbe861cb0: Status 404 returned error can't find the container with id ebcc497de0385cec2534cd64d952b4ad03cd22e6b4fbdd9d19e5f71cbe861cb0
Oct 30 03:59:00.471689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.471658    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:00.565602 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.565570    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" event={"ID":"de67c70c-bd18-4242-bed5-ab0ccecaa4fb","Type":"ContainerStarted","Data":"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"}
Oct 30 03:59:00.566009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.565622    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" event={"ID":"de67c70c-bd18-4242-bed5-ab0ccecaa4fb","Type":"ContainerStarted","Data":"5bf62cbfce5f13d6f1dcfe2573d57aa758bb3869cc8def9a1e6c39710558baf4"}
Oct 30 03:59:00.566009 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.565691    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:59:00.566947 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.566919    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" event={"ID":"c76c47f4-ab68-44b7-8da9-e30bbf304f11","Type":"ContainerStarted","Data":"d1ab40fbdc17b5f13fb41beb4e068c01552d2934f982c4f78ae7d36ebb5a1aa1"}
Oct 30 03:59:00.568400 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.568381    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" event={"ID":"f4b7ebc4-45da-47c3-88cb-c8647ed8a154","Type":"ContainerStarted","Data":"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"}
Oct 30 03:59:00.568503 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.568413    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" event={"ID":"f4b7ebc4-45da-47c3-88cb-c8647ed8a154","Type":"ContainerStarted","Data":"5539522619c70349f542144a77bae3457efa1e8738835ed1febbc27f27c0c950"}
Oct 30 03:59:00.568503 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.568463    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:59:00.569852 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.569808    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" event={"ID":"9341140b-f736-4637-84c5-9deea3281a85","Type":"ContainerStarted","Data":"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"}
Oct 30 03:59:00.569852 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.569853    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" event={"ID":"9341140b-f736-4637-84c5-9deea3281a85","Type":"ContainerStarted","Data":"ebcc497de0385cec2534cd64d952b4ad03cd22e6b4fbdd9d19e5f71cbe861cb0"}
Oct 30 03:59:00.569972 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.569955    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 03:59:00.603807 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.603752    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" podStartSLOduration=1.6037395220000001 podStartE2EDuration="1.603739522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:00.602738501 +0000 UTC m=+4698.826987779" watchObservedRunningTime="2026-10-30 03:59:00.603739522 +0000 UTC m=+4698.827988792"
Oct 30 03:59:00.644902 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.644859    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podStartSLOduration=1.6448452 podStartE2EDuration="1.6448452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:00.640792748 +0000 UTC m=+4698.865042025" watchObservedRunningTime="2026-10-30 03:59:00.6448452 +0000 UTC m=+4698.869094473"
Oct 30 03:59:00.683242 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.683146    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" podStartSLOduration=1.683071019 podStartE2EDuration="1.683071019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:00.681679647 +0000 UTC m=+4698.905928925" watchObservedRunningTime="2026-10-30 03:59:00.683071019 +0000 UTC m=+4698.907320299"
Oct 30 03:59:00.772423 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:00.772318    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:00.776860 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:59:00.776833    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1042c3a7_d26e_41a8_ac1f_abbdeacc3845.slice/crio-8a73272a72b58f52e637e2ac78ee6fcd61ed1b2723495b052b58d1b8d936559e WatchSource:0}: Error finding container 8a73272a72b58f52e637e2ac78ee6fcd61ed1b2723495b052b58d1b8d936559e: Status 404 returned error can't find the container with id 8a73272a72b58f52e637e2ac78ee6fcd61ed1b2723495b052b58d1b8d936559e
Oct 30 03:59:01.576414 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.576379    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" event={"ID":"c76c47f4-ab68-44b7-8da9-e30bbf304f11","Type":"ContainerStarted","Data":"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"}
Oct 30 03:59:01.576897 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.576507    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:59:01.578236 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.578210    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerStarted","Data":"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"}
Oct 30 03:59:01.578339 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.578258    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerStarted","Data":"8a73272a72b58f52e637e2ac78ee6fcd61ed1b2723495b052b58d1b8d936559e"}
Oct 30 03:59:01.615088 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.615040    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" podStartSLOduration=2.615028821 podStartE2EDuration="2.615028821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:01.614585534 +0000 UTC m=+4699.838834811" watchObservedRunningTime="2026-10-30 03:59:01.615028821 +0000 UTC m=+4699.839278097"
Oct 30 03:59:01.868656 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.868565    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:01.987937 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:01.987905    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:02.577418 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.577375    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:59:02.581314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.581289    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.584860 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.584806    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerStarted","Data":"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"}
Oct 30 03:59:02.585024 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.585004    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:02.585093 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.585058    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:02.586648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.586609    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.79:8083/ready\": dial tcp 10.133.0.79:8083: connect: connection refused"
Oct 30 03:59:02.594046 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.594027    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 03:59:02.605553 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.605532    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:59:02.659648 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.659585    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" podStartSLOduration=2.659567359 podStartE2EDuration="2.659567359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:02.657793422 +0000 UTC m=+4700.882042699" watchObservedRunningTime="2026-10-30 03:59:02.659567359 +0000 UTC m=+4700.883816637"
Oct 30 03:59:02.684770 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.684730    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.685120 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.685091    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.685258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.685229    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.685371 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.685353    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847dl\" (UniqueName: \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.768382 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.768338    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:02.786151 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786115    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.786323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786163    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847dl\" (UniqueName: \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.786323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786193    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.786323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786228    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.786837 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786798    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.787003 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.786984    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.788628 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.788607    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:02.808216 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:02.808191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847dl\" (UniqueName: \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:03.130410 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:03.130373    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:03.590375 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:03.590298    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:04.630712 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:59:04.630681    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:59:07.083468 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.083433    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:07.083971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.083638    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway" containerID="cri-o://a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d" gracePeriod=30
Oct 30 03:59:07.083971 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.083685    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway-opa" containerID="cri-o://321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320" gracePeriod=30
Oct 30 03:59:07.590378 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.590342    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:07.608570 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.608537    4528 generic.go:356] "Generic (PLEG): container finished" podID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerID="321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320" exitCode=0
Oct 30 03:59:07.608700 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:07.608606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerDied","Data":"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"}
Oct 30 03:59:08.320913 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.320885    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:08.437421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437330    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437421 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437392    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\" (UniqueName: \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437427    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437514    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437555    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437639 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437594    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\") pod \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\" (UID: \"1042c3a7-d26e-41a8-ac1f-abbdeacc3845\") "
Oct 30 03:59:08.437896 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437738    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:08.437999 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.437973    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:08.438056 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.438039    4528 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.438235 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.438209    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:08.439539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.439517    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "kube-api-access-jh62t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:08.439659 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.439642    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:08.439703 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.439666    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:08.444689 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.444665    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant" pod "1042c3a7-d26e-41a8-ac1f-abbdeacc3845" (UID: "1042c3a7-d26e-41a8-ac1f-abbdeacc3845"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:08.539035 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.538997    4528 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tenant\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.539035 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.539029    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh62t\" (UniqueName: \"kubernetes.io/projected/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-kube-api-access-jh62t\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.539035 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.539039    4528 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-serving-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.539267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.539049    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.539267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.539058    4528 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.539267 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.539066    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/1042c3a7-d26e-41a8-ac1f-abbdeacc3845-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 03:59:08.614168 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.614133    4528 generic.go:356] "Generic (PLEG): container finished" podID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerID="a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d" exitCode=0
Oct 30 03:59:08.614323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.614192    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"
Oct 30 03:59:08.614323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.614205    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerDied","Data":"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"}
Oct 30 03:59:08.614323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.614251    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr" event={"ID":"1042c3a7-d26e-41a8-ac1f-abbdeacc3845","Type":"ContainerDied","Data":"8a73272a72b58f52e637e2ac78ee6fcd61ed1b2723495b052b58d1b8d936559e"}
Oct 30 03:59:08.614323 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.614279    4528 scope.go:122] "RemoveContainer" containerID="321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"
Oct 30 03:59:08.622589 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.622572    4528 scope.go:122] "RemoveContainer" containerID="a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"
Oct 30 03:59:08.630547 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.630529    4528 scope.go:122] "RemoveContainer" containerID="321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"
Oct 30 03:59:08.630851 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:59:08.630812    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320\": container with ID starting with 321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320 not found: ID does not exist" containerID="321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"
Oct 30 03:59:08.630893 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.630860    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320"} err="failed to get container status \"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320\": rpc error: code = NotFound desc = could not find container \"321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320\": container with ID starting with 321c7d71be50c2e1f28b4724cfd33c75715c14755dcacfb315b90a8cbef3c320 not found: ID does not exist"
Oct 30 03:59:08.630893 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.630878    4528 scope.go:122] "RemoveContainer" containerID="a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"
Oct 30 03:59:08.631117 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:59:08.631098    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d\": container with ID starting with a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d not found: ID does not exist" containerID="a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"
Oct 30 03:59:08.631157 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.631123    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d"} err="failed to get container status \"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d\": rpc error: code = NotFound desc = could not find container \"a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d\": container with ID starting with a87fe09375f4d863b58b74fc3cef3c9e88f44f59364bf27abe4e266af60b831d not found: ID does not exist"
Oct 30 03:59:08.659852 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.659811    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:08.663576 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:08.663555    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-665876f97-dxzvr"]
Oct 30 03:59:09.892246 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:09.892215    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" path="/var/lib/kubelet/pods/1042c3a7-d26e-41a8-ac1f-abbdeacc3845/volumes"
Oct 30 03:59:11.060561 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.060525    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.085000 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.084969    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.161149 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.161115    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.161290 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.161218    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.173314 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.173286    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") DevicePath \"csi-7aa53e340b2b8925302958b156c025cf68a27065b5ecfbbdd417e244edeb1c66\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.398577 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.398546    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7e9a666e0ba128a61b3fe9d2d9fd641d359a39f007a46310717d51259c1b73d9/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.420258 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.420231    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.594803 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.594768    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:11.906214 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:11.906183    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 03:59:11.910743 ip-10-0-134-164 kubenswrapper[4528]: W1030 03:59:11.910708    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7ab986_ded9_4410_b8e9_161ae1835024.slice/crio-f34369d9faa09b1d7ef72a5ea3ac2440722379ecd79e97d8d7eb9cf4559f46b7 WatchSource:0}: Error finding container f34369d9faa09b1d7ef72a5ea3ac2440722379ecd79e97d8d7eb9cf4559f46b7: Status 404 returned error can't find the container with id f34369d9faa09b1d7ef72a5ea3ac2440722379ecd79e97d8d7eb9cf4559f46b7
Oct 30 03:59:12.638845 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:12.638794    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7a7ab986-ded9-4410-b8e9-161ae1835024","Type":"ContainerStarted","Data":"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"}
Oct 30 03:59:12.639368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:12.638876    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7a7ab986-ded9-4410-b8e9-161ae1835024","Type":"ContainerStarted","Data":"f34369d9faa09b1d7ef72a5ea3ac2440722379ecd79e97d8d7eb9cf4559f46b7"}
Oct 30 03:59:12.639368 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:12.638908    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 03:59:12.687409 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:12.687352    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=13.687338099 podStartE2EDuration="13.687338099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:12.676652428 +0000 UTC m=+4710.900901706" watchObservedRunningTime="2026-10-30 03:59:12.687338099 +0000 UTC m=+4710.911587375"
Oct 30 03:59:21.583692 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:21.583657    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 03:59:21.584707 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:21.584681    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:21.584864 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:21.584847    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 03:59:22.590925 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:22.590896    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 03:59:31.582688 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:31.582644    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:33.648539 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:33.648494    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:34.631161 ip-10-0-134-164 kubenswrapper[4528]: E1030 03:59:34.631127    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 03:59:39.380311 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:39.380274    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 03:59:41.582403 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:41.582359    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:43.647455 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:43.647415    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:51.583316 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:51.583281    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:59:53.647707 ip-10-0-134-164 kubenswrapper[4528]: I1030 03:59:53.647676    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:01.583002 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:01.582969    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 04:00:04.562383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:04.562351    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:04.632149 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:04.632116    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:00:05.121773 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.121733    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:05.122048 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.121988    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4a1f334f-1724-41dd-8081-08be2240cc79" containerName="otc-container" containerID="cri-o://9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7" gracePeriod=30
Oct 30 04:00:05.374921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.374858    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:05.427028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.426996    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\" (UniqueName: \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\") pod \"4a1f334f-1724-41dd-8081-08be2240cc79\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") "
Oct 30 04:00:05.427233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.427050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\") pod \"4a1f334f-1724-41dd-8081-08be2240cc79\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") "
Oct 30 04:00:05.427233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.427116    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\") pod \"4a1f334f-1724-41dd-8081-08be2240cc79\" (UID: \"4a1f334f-1724-41dd-8081-08be2240cc79\") "
Oct 30 04:00:05.427548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.427523    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal" pod "4a1f334f-1724-41dd-8081-08be2240cc79" (UID: "4a1f334f-1724-41dd-8081-08be2240cc79"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:05.429204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.429178    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs" pod "4a1f334f-1724-41dd-8081-08be2240cc79" (UID: "4a1f334f-1724-41dd-8081-08be2240cc79"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:05.429204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.429192    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl" pod "4a1f334f-1724-41dd-8081-08be2240cc79" (UID: "4a1f334f-1724-41dd-8081-08be2240cc79"). InnerVolumeSpecName "kube-api-access-qztnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:05.528223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.528187    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztnl\" (UniqueName: \"kubernetes.io/projected/4a1f334f-1724-41dd-8081-08be2240cc79-kube-api-access-qztnl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:05.528223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.528217    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1f334f-1724-41dd-8081-08be2240cc79-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:05.528223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.528226    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4a1f334f-1724-41dd-8081-08be2240cc79-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:05.874878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.874840    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a1f334f-1724-41dd-8081-08be2240cc79" containerID="9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7" exitCode=0
Oct 30 04:00:05.875304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.874884    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4a1f334f-1724-41dd-8081-08be2240cc79","Type":"ContainerDied","Data":"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"}
Oct 30 04:00:05.875304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.874901    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:05.875304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.874968    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4a1f334f-1724-41dd-8081-08be2240cc79","Type":"ContainerDied","Data":"f4ee5fa15c44ceee6dfcf210d999e09284ed9f9eb388aea25a18092cc91958b9"}
Oct 30 04:00:05.875304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.874998    4528 scope.go:122] "RemoveContainer" containerID="9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"
Oct 30 04:00:05.883642 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.883622    4528 scope.go:122] "RemoveContainer" containerID="9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"
Oct 30 04:00:05.883933 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:05.883913    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7\": container with ID starting with 9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7 not found: ID does not exist" containerID="9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"
Oct 30 04:00:05.884000 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.883946    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7"} err="failed to get container status \"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7\": rpc error: code = NotFound desc = could not find container \"9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7\": container with ID starting with 9db69e314d4d4f098dc793296de806d67868b825eac5583aadb31936253566a7 not found: ID does not exist"
Oct 30 04:00:05.920073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.920038    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:05.922498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.922472    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:05.968665 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.968629    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:05.969182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969127    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway-opa"
Oct 30 04:00:05.969182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969153    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway-opa"
Oct 30 04:00:05.969182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969174    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a1f334f-1724-41dd-8081-08be2240cc79" containerName="otc-container"
Oct 30 04:00:05.969182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1f334f-1724-41dd-8081-08be2240cc79" containerName="otc-container"
Oct 30 04:00:05.969482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969225    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway"
Oct 30 04:00:05.969482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969232    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway"
Oct 30 04:00:05.969482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969323    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway"
Oct 30 04:00:05.969482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969335    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1042c3a7-d26e-41a8-ac1f-abbdeacc3845" containerName="tempo-gateway-opa"
Oct 30 04:00:05.969482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.969346    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1f334f-1724-41dd-8081-08be2240cc79" containerName="otc-container"
Oct 30 04:00:05.973757 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.973735    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:05.988670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.988644    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\""
Oct 30 04:00:05.988993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.988974    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:00:05.989105 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:05.989002    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8l25s\""
Oct 30 04:00:06.003516 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.003484    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:06.032209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.032180    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.032369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.032233    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.032420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.032394    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrks\" (UniqueName: \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.133527 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.133494    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.133684 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.133558    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrks\" (UniqueName: \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.133684 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.133595    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.134267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.134244    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.136024 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.135992    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.155046 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.155021    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrks\" (UniqueName: \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\") pod \"data-science-collector-collector-1\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.285762 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.285720    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:06.582498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.582417    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:06.587191 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:06.587155    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfdc17d_d9f5_42b6_b149_b5b1f7bc50a3.slice/crio-313c1dc1b60921fef872c7f0d9d9c59232f48c6ffae3ae8c76632dbe88fb604f WatchSource:0}: Error finding container 313c1dc1b60921fef872c7f0d9d9c59232f48c6ffae3ae8c76632dbe88fb604f: Status 404 returned error can't find the container with id 313c1dc1b60921fef872c7f0d9d9c59232f48c6ffae3ae8c76632dbe88fb604f
Oct 30 04:00:06.589412 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.589377    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:00:06.589530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.589453    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:00:06.682945 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.682910    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:06.881878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.881851    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3","Type":"ContainerStarted","Data":"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"}
Oct 30 04:00:06.882239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.881901    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3","Type":"ContainerStarted","Data":"313c1dc1b60921fef872c7f0d9d9c59232f48c6ffae3ae8c76632dbe88fb604f"}
Oct 30 04:00:06.920513 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:06.920463    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.920450489 podStartE2EDuration="1.920450489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:06.919164555 +0000 UTC m=+4765.143413815" watchObservedRunningTime="2026-10-30 04:00:06.920450489 +0000 UTC m=+4765.144699765"
Oct 30 04:00:07.892147 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:07.892116    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1f334f-1724-41dd-8081-08be2240cc79" path="/var/lib/kubelet/pods/4a1f334f-1724-41dd-8081-08be2240cc79/volumes"
Oct 30 04:00:08.540631 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:08.540596    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:08.602328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:08.602290    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:08.724385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:08.724352    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:08.841243 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:08.841152    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:08.890453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:08.890416    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" containerName="otc-container" containerID="cri-o://96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639" gracePeriod=30
Oct 30 04:00:09.110526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.109945    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:09.127311 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.127278    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:09.127518 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.127496    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo" containerID="cri-o://8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090" gracePeriod=30
Oct 30 04:00:09.154186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.154153    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 04:00:09.154406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.154380    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" containerID="cri-o://6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb" gracePeriod=30
Oct 30 04:00:09.160263 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.159970    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 04:00:09.160263 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.160136    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" podUID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" containerName="tempo" containerID="cri-o://5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5" gracePeriod=30
Oct 30 04:00:09.167487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.166801    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 04:00:09.167685 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.167663    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo" containerID="cri-o://650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2" gracePeriod=30
Oct 30 04:00:09.174148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.174122    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 04:00:09.174330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.174301    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo" containerID="cri-o://e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485" gracePeriod=30
Oct 30 04:00:09.194251 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.194227    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:09.226208 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.226171    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:09.262561 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.262529    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\") pod \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") "
Oct 30 04:00:09.262749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.262589    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\") pod \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") "
Oct 30 04:00:09.262749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.262654    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\" (UniqueName: \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\") pod \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\" (UID: \"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3\") "
Oct 30 04:00:09.263028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.263001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal" pod "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" (UID: "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:09.264723 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.264700    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs" pod "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" (UID: "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:09.264803 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.264783    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks" pod "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" (UID: "2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3"). InnerVolumeSpecName "kube-api-access-vqrks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:09.363526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.363466    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrks\" (UniqueName: \"kubernetes.io/projected/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-kube-api-access-vqrks\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.363526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.363494    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.363526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.363504    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.515749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.515726    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 04:00:09.666381 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666350    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\") pod \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") "
Oct 30 04:00:09.666556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\") pod \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") "
Oct 30 04:00:09.666556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666444    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\" (UniqueName: \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\") pod \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") "
Oct 30 04:00:09.666556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\") pod \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") "
Oct 30 04:00:09.666556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666541    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\") pod \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\" (UID: \"f4b7ebc4-45da-47c3-88cb-c8647ed8a154\") "
Oct 30 04:00:09.666850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage" pod "f4b7ebc4-45da-47c3-88cb-c8647ed8a154" (UID: "f4b7ebc4-45da-47c3-88cb-c8647ed8a154"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:09.666912 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666893    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle" pod "f4b7ebc4-45da-47c3-88cb-c8647ed8a154" (UID: "f4b7ebc4-45da-47c3-88cb-c8647ed8a154"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:09.666955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.666899    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf" pod "f4b7ebc4-45da-47c3-88cb-c8647ed8a154" (UID: "f4b7ebc4-45da-47c3-88cb-c8647ed8a154"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:09.668674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.668647    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg" pod "f4b7ebc4-45da-47c3-88cb-c8647ed8a154" (UID: "f4b7ebc4-45da-47c3-88cb-c8647ed8a154"). InnerVolumeSpecName "kube-api-access-wstrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:09.668674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.668655    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls" pod "f4b7ebc4-45da-47c3-88cb-c8647ed8a154" (UID: "f4b7ebc4-45da-47c3-88cb-c8647ed8a154"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:09.695070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.695045    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"]
Oct 30 04:00:09.699221 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.699196    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"]
Oct 30 04:00:09.767382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.767348    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.767382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.767378    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.767382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.767388    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.767620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.767399    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstrg\" (UniqueName: \"kubernetes.io/projected/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-kube-api-access-wstrg\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.767620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.767408    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f4b7ebc4-45da-47c3-88cb-c8647ed8a154-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:09.893458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.893418    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33452438-7732-4080-b8e4-77295c645cb1" path="/var/lib/kubelet/pods/33452438-7732-4080-b8e4-77295c645cb1/volumes"
Oct 30 04:00:09.902949 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.902914    4528 generic.go:356] "Generic (PLEG): container finished" podID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" containerID="5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5" exitCode=0
Oct 30 04:00:09.903124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.902978    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"
Oct 30 04:00:09.903124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.902998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" event={"ID":"f4b7ebc4-45da-47c3-88cb-c8647ed8a154","Type":"ContainerDied","Data":"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"}
Oct 30 04:00:09.903124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.903060    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr" event={"ID":"f4b7ebc4-45da-47c3-88cb-c8647ed8a154","Type":"ContainerDied","Data":"5539522619c70349f542144a77bae3457efa1e8738835ed1febbc27f27c0c950"}
Oct 30 04:00:09.903124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.903090    4528 scope.go:122] "RemoveContainer" containerID="5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"
Oct 30 04:00:09.908229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.908204    4528 generic.go:356] "Generic (PLEG): container finished" podID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" containerID="96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639" exitCode=0
Oct 30 04:00:09.908366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.908255    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:09.908366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.908258    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3","Type":"ContainerDied","Data":"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"}
Oct 30 04:00:09.908366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.908360    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3","Type":"ContainerDied","Data":"313c1dc1b60921fef872c7f0d9d9c59232f48c6ffae3ae8c76632dbe88fb604f"}
Oct 30 04:00:09.914739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.914722    4528 scope.go:122] "RemoveContainer" containerID="5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"
Oct 30 04:00:09.915105 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:09.915086    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5\": container with ID starting with 5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5 not found: ID does not exist" containerID="5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"
Oct 30 04:00:09.915195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.915112    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5"} err="failed to get container status \"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5\": rpc error: code = NotFound desc = could not find container \"5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5\": container with ID starting with 5e777e7f88f710878f0235dba5defec05478fb28f51e9459bb00da7deef564b5 not found: ID does not exist"
Oct 30 04:00:09.915195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.915129    4528 scope.go:122] "RemoveContainer" containerID="96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"
Oct 30 04:00:09.922349 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.922331    4528 scope.go:122] "RemoveContainer" containerID="96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"
Oct 30 04:00:09.922592 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:09.922572    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639\": container with ID starting with 96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639 not found: ID does not exist" containerID="96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"
Oct 30 04:00:09.922642 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.922599    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639"} err="failed to get container status \"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639\": rpc error: code = NotFound desc = could not find container \"96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639\": container with ID starting with 96c48cf498b0091d8223d81209613b36aa3abc188a04ade6cc070fdc7d141639 not found: ID does not exist"
Oct 30 04:00:09.946721 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.946697    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:09.950951 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.950932    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:09.980923 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.980896    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 04:00:09.984905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:09.984881    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-57bb9fc7d4-n9cgr"]
Oct 30 04:00:11.583715 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:11.583666    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:11.584139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:11.584070    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:11.891892 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:11.891861    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" path="/var/lib/kubelet/pods/2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3/volumes"
Oct 30 04:00:11.892265 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:11.892253    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" path="/var/lib/kubelet/pods/f4b7ebc4-45da-47c3-88cb-c8647ed8a154/volumes"
Oct 30 04:00:12.590186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:12.590151    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:13.107760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.107738    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 04:00:13.197660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.197582    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\") pod \"9341140b-f736-4637-84c5-9deea3281a85\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") "
Oct 30 04:00:13.197660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.197642    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\") pod \"9341140b-f736-4637-84c5-9deea3281a85\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") "
Oct 30 04:00:13.197919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.197668    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\" (UniqueName: \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\") pod \"9341140b-f736-4637-84c5-9deea3281a85\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") "
Oct 30 04:00:13.197919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.197701    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\") pod \"9341140b-f736-4637-84c5-9deea3281a85\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") "
Oct 30 04:00:13.197919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.197865    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\") pod \"9341140b-f736-4637-84c5-9deea3281a85\" (UID: \"9341140b-f736-4637-84c5-9deea3281a85\") "
Oct 30 04:00:13.198129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.198062    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle" pod "9341140b-f736-4637-84c5-9deea3281a85" (UID: "9341140b-f736-4637-84c5-9deea3281a85"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:13.198241 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.198213    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf" pod "9341140b-f736-4637-84c5-9deea3281a85" (UID: "9341140b-f736-4637-84c5-9deea3281a85"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:13.198279 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.198218    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage" pod "9341140b-f736-4637-84c5-9deea3281a85" (UID: "9341140b-f736-4637-84c5-9deea3281a85"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:13.199911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.199884    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls" pod "9341140b-f736-4637-84c5-9deea3281a85" (UID: "9341140b-f736-4637-84c5-9deea3281a85"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:13.199991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.199930    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262" pod "9341140b-f736-4637-84c5-9deea3281a85" (UID: "9341140b-f736-4637-84c5-9deea3281a85"). InnerVolumeSpecName "kube-api-access-hs262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:13.298765 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.298735    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:13.298765 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.298762    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:13.298972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.298776    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs262\" (UniqueName: \"kubernetes.io/projected/9341140b-f736-4637-84c5-9deea3281a85-kube-api-access-hs262\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:13.298972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.298785    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/9341140b-f736-4637-84c5-9deea3281a85-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:13.298972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.298794    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/9341140b-f736-4637-84c5-9deea3281a85-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:13.648081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.648048    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:13.928659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.928626    4528 generic.go:356] "Generic (PLEG): container finished" podID="9341140b-f736-4637-84c5-9deea3281a85" containerID="6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb" exitCode=0
Oct 30 04:00:13.928793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.928685    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"
Oct 30 04:00:13.928793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.928704    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" event={"ID":"9341140b-f736-4637-84c5-9deea3281a85","Type":"ContainerDied","Data":"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"}
Oct 30 04:00:13.928793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.928753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l" event={"ID":"9341140b-f736-4637-84c5-9deea3281a85","Type":"ContainerDied","Data":"ebcc497de0385cec2534cd64d952b4ad03cd22e6b4fbdd9d19e5f71cbe861cb0"}
Oct 30 04:00:13.928793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.928774    4528 scope.go:122] "RemoveContainer" containerID="6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"
Oct 30 04:00:13.936631 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.936614    4528 scope.go:122] "RemoveContainer" containerID="6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"
Oct 30 04:00:13.936933 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:13.936914    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb\": container with ID starting with 6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb not found: ID does not exist" containerID="6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"
Oct 30 04:00:13.936999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.936941    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb"} err="failed to get container status \"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb\": rpc error: code = NotFound desc = could not find container \"6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb\": container with ID starting with 6a5af2aec988106087b92c9eb52ffaedbab16554318fc0d68703f0b13f54a2fb not found: ID does not exist"
Oct 30 04:00:13.972908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.972876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 04:00:13.982572 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:13.982550    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-589b765f47-nft5l"]
Oct 30 04:00:15.892371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:15.892338    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9341140b-f736-4637-84c5-9deea3281a85" path="/var/lib/kubelet/pods/9341140b-f736-4637-84c5-9deea3281a85/volumes"
Oct 30 04:00:21.582560 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:21.582523    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:22.631515 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:22.631479    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:23.647178 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:23.647137    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:31.364727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.364699    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:31.552891 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.552790    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\") pod \"7a7ab986-ded9-4410-b8e9-161ae1835024\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") "
Oct 30 04:00:31.553042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.552905    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\") pod \"7a7ab986-ded9-4410-b8e9-161ae1835024\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") "
Oct 30 04:00:31.553042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.552941    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\" (UniqueName: \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\") pod \"7a7ab986-ded9-4410-b8e9-161ae1835024\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") "
Oct 30 04:00:31.553042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.552958    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\") pod \"7a7ab986-ded9-4410-b8e9-161ae1835024\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") "
Oct 30 04:00:31.553148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.553043    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"7a7ab986-ded9-4410-b8e9-161ae1835024\" (UID: \"7a7ab986-ded9-4410-b8e9-161ae1835024\") "
Oct 30 04:00:31.553242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.553213    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle" pod "7a7ab986-ded9-4410-b8e9-161ae1835024" (UID: "7a7ab986-ded9-4410-b8e9-161ae1835024"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:31.553437 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.553412    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf" pod "7a7ab986-ded9-4410-b8e9-161ae1835024" (UID: "7a7ab986-ded9-4410-b8e9-161ae1835024"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:31.555094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.555065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls" pod "7a7ab986-ded9-4410-b8e9-161ae1835024" (UID: "7a7ab986-ded9-4410-b8e9-161ae1835024"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:31.555198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.555173    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl" pod "7a7ab986-ded9-4410-b8e9-161ae1835024" (UID: "7a7ab986-ded9-4410-b8e9-161ae1835024"). InnerVolumeSpecName "kube-api-access-847dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:31.555830 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.555802    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879" pod "7a7ab986-ded9-4410-b8e9-161ae1835024" (UID: "7a7ab986-ded9-4410-b8e9-161ae1835024"). InnerVolumeSpecName "pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:00:31.582771 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.582740    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:31.582885 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.582862    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 04:00:31.654338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.654299    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:31.654338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.654330    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:31.654338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.654343    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847dl\" (UniqueName: \"kubernetes.io/projected/7a7ab986-ded9-4410-b8e9-161ae1835024-kube-api-access-847dl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:31.654584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.654353    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7a7ab986-ded9-4410-b8e9-161ae1835024-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:31.654584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.654381    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:00:31.663716 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.663697    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:00:31.755086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:31.755055    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-7aa53e340b2b8925302958b156c025cf68a27065b5ecfbbdd417e244edeb1c66\""
Oct 30 04:00:32.002971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.002930    4528 generic.go:356] "Generic (PLEG): container finished" podID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerID="8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090" exitCode=0
Oct 30 04:00:32.003143 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.002999    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:32.003143 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.003014    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7a7ab986-ded9-4410-b8e9-161ae1835024","Type":"ContainerDied","Data":"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"}
Oct 30 04:00:32.003143 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.003074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7a7ab986-ded9-4410-b8e9-161ae1835024","Type":"ContainerDied","Data":"f34369d9faa09b1d7ef72a5ea3ac2440722379ecd79e97d8d7eb9cf4559f46b7"}
Oct 30 04:00:32.003143 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.003096    4528 scope.go:122] "RemoveContainer" containerID="8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"
Oct 30 04:00:32.011001 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.010984    4528 scope.go:122] "RemoveContainer" containerID="8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"
Oct 30 04:00:32.011284 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:32.011264    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090\": container with ID starting with 8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090 not found: ID does not exist" containerID="8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"
Oct 30 04:00:32.011357 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.011289    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090"} err="failed to get container status \"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090\": rpc error: code = NotFound desc = could not find container \"8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090\": container with ID starting with 8075ae5bf3999a62e3a08bf4702e41d21a0789ac2ee1a87dc377dc2c462c0090 not found: ID does not exist"
Oct 30 04:00:32.047419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.047389    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:32.057260 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.057236    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:32.629871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.629809    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:00:32.630325 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:32.629940    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 04:00:33.891595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:33.891563    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" path="/var/lib/kubelet/pods/7a7ab986-ded9-4410-b8e9-161ae1835024/volumes"
Oct 30 04:00:34.631047 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:34.631013    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:00:36.093698 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:36.093666    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:38.429189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:38.429151    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:39.342235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.342211    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 04:00:39.344998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.344977    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 04:00:39.408468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408438    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\" (UniqueName: \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\") pod \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") "
Oct 30 04:00:39.408640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408489    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\") pod \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") "
Oct 30 04:00:39.408640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\") pod \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") "
Oct 30 04:00:39.408640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408556    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\") pod \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") "
Oct 30 04:00:39.408640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408593    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\") pod \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") "
Oct 30 04:00:39.408640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408619    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\") pod \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") "
Oct 30 04:00:39.408904 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408650    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\") pod \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") "
Oct 30 04:00:39.408904 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\") pod \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") "
Oct 30 04:00:39.409017 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.408958    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage" pod "de67c70c-bd18-4242-bed5-ab0ccecaa4fb" (UID: "de67c70c-bd18-4242-bed5-ab0ccecaa4fb"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:39.409078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409038    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\") pod \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\" (UID: \"c76c47f4-ab68-44b7-8da9-e30bbf304f11\") "
Oct 30 04:00:39.409140 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409105    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf" pod "de67c70c-bd18-4242-bed5-ab0ccecaa4fb" (UID: "de67c70c-bd18-4242-bed5-ab0ccecaa4fb"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:39.409140 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409113    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle" pod "de67c70c-bd18-4242-bed5-ab0ccecaa4fb" (UID: "de67c70c-bd18-4242-bed5-ab0ccecaa4fb"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:39.409140 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409114    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle" pod "c76c47f4-ab68-44b7-8da9-e30bbf304f11" (UID: "c76c47f4-ab68-44b7-8da9-e30bbf304f11"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:39.409425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409401    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.409556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409433    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.409556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409452    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.409556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409465    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.409556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage" pod "c76c47f4-ab68-44b7-8da9-e30bbf304f11" (UID: "c76c47f4-ab68-44b7-8da9-e30bbf304f11"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:39.409556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.409416    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf" pod "c76c47f4-ab68-44b7-8da9-e30bbf304f11" (UID: "c76c47f4-ab68-44b7-8da9-e30bbf304f11"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:39.410837 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.410798    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk" pod "c76c47f4-ab68-44b7-8da9-e30bbf304f11" (UID: "c76c47f4-ab68-44b7-8da9-e30bbf304f11"). InnerVolumeSpecName "kube-api-access-sjbmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:39.411353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.411326    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls" pod "c76c47f4-ab68-44b7-8da9-e30bbf304f11" (UID: "c76c47f4-ab68-44b7-8da9-e30bbf304f11"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:39.411436 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.411324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls" pod "de67c70c-bd18-4242-bed5-ab0ccecaa4fb" (UID: "de67c70c-bd18-4242-bed5-ab0ccecaa4fb"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:39.476001 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.475919    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:39.509986 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.509958    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\" (UniqueName: \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\") pod \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\" (UID: \"de67c70c-bd18-4242-bed5-ab0ccecaa4fb\") "
Oct 30 04:00:39.510131 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.510118    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjbmk\" (UniqueName: \"kubernetes.io/projected/c76c47f4-ab68-44b7-8da9-e30bbf304f11-kube-api-access-sjbmk\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.510173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.510133    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.510173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.510143    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.510173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.510152    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.510173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.510160    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/c76c47f4-ab68-44b7-8da9-e30bbf304f11-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:39.511997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.511972    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb" pod "de67c70c-bd18-4242-bed5-ab0ccecaa4fb" (UID: "de67c70c-bd18-4242-bed5-ab0ccecaa4fb"). InnerVolumeSpecName "kube-api-access-k2nlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:39.610622 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:39.610586    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2nlb\" (UniqueName: \"kubernetes.io/projected/de67c70c-bd18-4242-bed5-ab0ccecaa4fb-kube-api-access-k2nlb\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:40.037931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.037898    4528 generic.go:356] "Generic (PLEG): container finished" podID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerID="650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2" exitCode=137
Oct 30 04:00:40.038127 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.037943    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"
Oct 30 04:00:40.038127 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.037978    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" event={"ID":"de67c70c-bd18-4242-bed5-ab0ccecaa4fb","Type":"ContainerDied","Data":"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"}
Oct 30 04:00:40.038127 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.038032    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x" event={"ID":"de67c70c-bd18-4242-bed5-ab0ccecaa4fb","Type":"ContainerDied","Data":"5bf62cbfce5f13d6f1dcfe2573d57aa758bb3869cc8def9a1e6c39710558baf4"}
Oct 30 04:00:40.038127 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.038059    4528 scope.go:122] "RemoveContainer" containerID="650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"
Oct 30 04:00:40.039339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.039313    4528 generic.go:356] "Generic (PLEG): container finished" podID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerID="e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485" exitCode=137
Oct 30 04:00:40.039475 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.039360    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"
Oct 30 04:00:40.039475 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.039408    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" event={"ID":"c76c47f4-ab68-44b7-8da9-e30bbf304f11","Type":"ContainerDied","Data":"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"}
Oct 30 04:00:40.039475 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.039442    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt" event={"ID":"c76c47f4-ab68-44b7-8da9-e30bbf304f11","Type":"ContainerDied","Data":"d1ab40fbdc17b5f13fb41beb4e068c01552d2934f982c4f78ae7d36ebb5a1aa1"}
Oct 30 04:00:40.046184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.046167    4528 scope.go:122] "RemoveContainer" containerID="650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"
Oct 30 04:00:40.046449 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:40.046432    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2\": container with ID starting with 650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2 not found: ID does not exist" containerID="650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"
Oct 30 04:00:40.046509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.046458    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2"} err="failed to get container status \"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2\": rpc error: code = NotFound desc = could not find container \"650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2\": container with ID starting with 650eeae93b6c56566feaaefdf85eff730691434f328d4baef164bbc6434ac2e2 not found: ID does not exist"
Oct 30 04:00:40.046509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.046475    4528 scope.go:122] "RemoveContainer" containerID="e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"
Oct 30 04:00:40.053633 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.053608    4528 scope.go:122] "RemoveContainer" containerID="e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"
Oct 30 04:00:40.053909 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:40.053892    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485\": container with ID starting with e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485 not found: ID does not exist" containerID="e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"
Oct 30 04:00:40.053966 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.053916    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485"} err="failed to get container status \"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485\": rpc error: code = NotFound desc = could not find container \"e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485\": container with ID starting with e7440cfd6fb5c8c7e3b853fe033b3115b4400faed6a6b68a3bcfbe648169a485 not found: ID does not exist"
Oct 30 04:00:40.081722 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.081695    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 04:00:40.091605 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.091582    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-7fc7cd84b4-j8qkt"]
Oct 30 04:00:40.116360 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.116333    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 04:00:40.121049 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:40.121028    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-559fbdfbdf-qjt8x"]
Oct 30 04:00:41.865442 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:41.865406    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:41.892206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:41.892182    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" path="/var/lib/kubelet/pods/c76c47f4-ab68-44b7-8da9-e30bbf304f11/volumes"
Oct 30 04:00:41.892594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:41.892580    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" path="/var/lib/kubelet/pods/de67c70c-bd18-4242-bed5-ab0ccecaa4fb/volumes"
Oct 30 04:00:41.985578 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:41.985543    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:48.516372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516338    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"]
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516598    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516608    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516624    4528 cpu_manager.go:414] "containerMap: removing container" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516629    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516637    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516642    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516648    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516653    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516664    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" containerName="otc-container"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516669    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" containerName="otc-container"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516676    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516681    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516723    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9341140b-f736-4637-84c5-9deea3281a85" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516734    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="de67c70c-bd18-4242-bed5-ab0ccecaa4fb" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516739    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b7ebc4-45da-47c3-88cb-c8647ed8a154" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516745    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7ab986-ded9-4410-b8e9-161ae1835024" containerName="tempo"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516751    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfdc17d-d9f5-42b6-b149-b5b1f7bc50a3" containerName="otc-container"
Oct 30 04:00:48.516767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.516758    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76c47f4-ab68-44b7-8da9-e30bbf304f11" containerName="tempo"
Oct 30 04:00:48.521494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.521475    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:48.534126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.534108    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:00:48.543940 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.543920    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:00:48.544098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.544086    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:00:48.545246 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.545224    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"]
Oct 30 04:00:48.582625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.582594    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4d4m\" (UniqueName: \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\") pod \"fake-gcs-bucket-creator\" (UID: \"f412fc54-b812-44a7-8d20-fbd6c87b4cf5\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:48.683832 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.683785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4d4m\" (UniqueName: \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\") pod \"fake-gcs-bucket-creator\" (UID: \"f412fc54-b812-44a7-8d20-fbd6c87b4cf5\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:48.705098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.705074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4d4m\" (UniqueName: \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\") pod \"fake-gcs-bucket-creator\" (UID: \"f412fc54-b812-44a7-8d20-fbd6c87b4cf5\") " pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:48.831931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:48.831849    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:49.122722 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:49.122646    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"]
Oct 30 04:00:50.079418 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:50.079385    4528 generic.go:356] "Generic (PLEG): container finished" podID="f412fc54-b812-44a7-8d20-fbd6c87b4cf5" containerID="f760d209ce7c39adbac2b42de6213a74bec2704dde509c2b249c08b8673f8d77" exitCode=0
Oct 30 04:00:50.079796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:50.079420    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"f412fc54-b812-44a7-8d20-fbd6c87b4cf5","Type":"ContainerDied","Data":"f760d209ce7c39adbac2b42de6213a74bec2704dde509c2b249c08b8673f8d77"}
Oct 30 04:00:50.079796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:50.079460    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"f412fc54-b812-44a7-8d20-fbd6c87b4cf5","Type":"ContainerStarted","Data":"8a0383bc8abb4d0913f26e31bbb906d60bfdcf4dca8e954d75c031f97ab9af7a"}
Oct 30 04:00:51.208490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:51.208463    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:51.301857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:51.301799    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\" (UniqueName: \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\") pod \"f412fc54-b812-44a7-8d20-fbd6c87b4cf5\" (UID: \"f412fc54-b812-44a7-8d20-fbd6c87b4cf5\") "
Oct 30 04:00:51.303956 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:51.303920    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m" pod "f412fc54-b812-44a7-8d20-fbd6c87b4cf5" (UID: "f412fc54-b812-44a7-8d20-fbd6c87b4cf5"). InnerVolumeSpecName "kube-api-access-p4d4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:51.402694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:51.402650    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4d4m\" (UniqueName: \"kubernetes.io/projected/f412fc54-b812-44a7-8d20-fbd6c87b4cf5-kube-api-access-p4d4m\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.775012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:51.774914    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:52.087697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:52.087612    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-bucket-creator"
Oct 30 04:00:52.087697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:52.087632    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-bucket-creator" event={"ID":"f412fc54-b812-44a7-8d20-fbd6c87b4cf5","Type":"ContainerDied","Data":"8a0383bc8abb4d0913f26e31bbb906d60bfdcf4dca8e954d75c031f97ab9af7a"}
Oct 30 04:00:52.087697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:52.087672    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a0383bc8abb4d0913f26e31bbb906d60bfdcf4dca8e954d75c031f97ab9af7a"
Oct 30 04:00:53.754338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:53.754247    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:53.871590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:53.871541    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:53.992254 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:53.992223    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:54.478577 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.478538    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:54.478991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.478960    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f412fc54-b812-44a7-8d20-fbd6c87b4cf5" containerName="create-bucket"
Oct 30 04:00:54.479147 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.479130    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f412fc54-b812-44a7-8d20-fbd6c87b4cf5" containerName="create-bucket"
Oct 30 04:00:54.479330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.479317    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f412fc54-b812-44a7-8d20-fbd6c87b4cf5" containerName="create-bucket"
Oct 30 04:00:54.482421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.482401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.508323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.508284    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:54.528568 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.528533    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8c79\" (UniqueName: \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.528714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.528609    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.528714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.528641    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.532044 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.532018    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:54.577301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.577268    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:00:54.577458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.577369    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-vv788\""
Oct 30 04:00:54.577506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.577457    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:00:54.577542 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.577515    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\""
Oct 30 04:00:54.577598 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.577582    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:00:54.629778 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.629747    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8c79\" (UniqueName: \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.629998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.629807    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.629998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.629850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.630626 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.630602    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.632255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.632234    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.650458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.650422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8c79\" (UniqueName: \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\") pod \"data-science-collector-collector-1\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:54.654652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.654628    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:54.775752 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.775668    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:54.798518 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:54.798494    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:00:55.093417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:55.093306    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:00:55.098562 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:55.098519    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1327d07c_a646_455f_9f6f_05ff43c46b3a.slice/crio-08b0263a2afed035f9a7bf09c2b69c7d053fc53ae10f9d6aff2a76e6fb6b3f2a WatchSource:0}: Error finding container 08b0263a2afed035f9a7bf09c2b69c7d053fc53ae10f9d6aff2a76e6fb6b3f2a: Status 404 returned error can't find the container with id 08b0263a2afed035f9a7bf09c2b69c7d053fc53ae10f9d6aff2a76e6fb6b3f2a
Oct 30 04:00:55.100873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:55.100843    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:00:55.100973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:55.100916    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:00:56.103944 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:56.103907    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1327d07c-a646-455f-9f6f-05ff43c46b3a","Type":"ContainerStarted","Data":"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"}
Oct 30 04:00:56.104332 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:56.103962    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1327d07c-a646-455f-9f6f-05ff43c46b3a","Type":"ContainerStarted","Data":"08b0263a2afed035f9a7bf09c2b69c7d053fc53ae10f9d6aff2a76e6fb6b3f2a"}
Oct 30 04:00:56.140500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:56.140450    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.14043614 podStartE2EDuration="2.14043614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:56.140014287 +0000 UTC m=+4814.364263592" watchObservedRunningTime="2026-10-30 04:00:56.14043614 +0000 UTC m=+4814.364685416"
Oct 30 04:00:57.385898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.385860    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:00:57.389224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.389201    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.416698 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.416677    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-hk87b\""
Oct 30 04:00:57.416874 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.416718    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\""
Oct 30 04:00:57.416874 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.416833    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\""
Oct 30 04:00:57.416874 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.416850    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\""
Oct 30 04:00:57.418214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.418160    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:00:57.442857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.442791    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:57.449021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.448995    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:57.452268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.452248    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.453452 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.453427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.453562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.453482    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cc77\" (UniqueName: \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.453628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.453583    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.453681 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.453640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.453749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.453731    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.475754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.475724    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:57.507076 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.507047    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\""
Oct 30 04:00:57.507263 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.507162    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 04:00:57.554558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554529    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.554743 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554570    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.554743 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554596    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cc77\" (UniqueName: \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.554743 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554699    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.554931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554748    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.554931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554798    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.554931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554840    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.554931 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554868    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.555146 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554947    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.555146 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.554995    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4jj\" (UniqueName: \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.555271 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.555209    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.555459 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.555436    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.555641 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.555620    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.557062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.557043    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.578019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.577980    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:57.578778 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.578759    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cc77\" (UniqueName: \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\") pod \"tempo-data-science-tempostack-distributor-5f59c684f-9fxbw\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.586538 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.586513    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:00:57.590078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.590058    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.607037 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.607015    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\""
Oct 30 04:00:57.613341 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.613312    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:00:57.636631 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.636550    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:00:57.640366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.640342    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.655894 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.655867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.655997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.655902    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.655997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.655921    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.655997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.655937    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656023    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.656111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.656171 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656105    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.656171 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656155    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.656233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656204    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.656268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656232    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4jj\" (UniqueName: \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.656308 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656269    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k96z\" (UniqueName: \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656349 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656328    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656399 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656369    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklsk\" (UniqueName: \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.656466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656415    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656574 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656553    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.656652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656628    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656709 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656693    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.656760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.656914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.656890    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.658543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.658523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.658781 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.658762    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.665258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.665236    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:00:57.678035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.678000    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4jj\" (UniqueName: \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:57.696802 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.696769    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:57.701564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.701541    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:57.706722 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.706689    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:00:57.707260 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.707236    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\""
Oct 30 04:00:57.710390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.710245    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.724369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.724347    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\""
Oct 30 04:00:57.732991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.732687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:00:57.758504 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758469    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmkr\" (UniqueName: \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.758674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758517    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.758674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.758674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklsk\" (UniqueName: \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.758674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758613    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.758674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758657    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758740    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758777    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758812    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758870    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758900    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.758963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758958    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.759358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.758992    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.759358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.759032    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.759358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.759071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k96z\" (UniqueName: \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.759358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.759102    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.760219 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.760195    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.761137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.761112    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.761592 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.761565    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.763086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.761961    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.763086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.762375    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.763086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.762528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.763086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.762906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.768660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.768600    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.768660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.768633    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.770717 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.770688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.780353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.780303    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklsk\" (UniqueName: \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\") pod \"tempo-data-science-tempostack-querier-65746454dc-gnhfr\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:57.783990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.783903    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k96z\" (UniqueName: \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\") pod \"tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.808161 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.808130    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:57.860256 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860224    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860268    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860293    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmkr\" (UniqueName: \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860535    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860638 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860615    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.860951 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.860926    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.861224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.861207    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.861289 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.861266    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.863506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.863475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.863598 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.863520    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.881320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.881299    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmkr\" (UniqueName: \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\") pod \"tempo-data-science-tempostack-compactor-759575fd59-52wzp\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:57.902091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.902026    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:57.925969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.925941    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:57.952533 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:57.952447    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:00:58.028050 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.028022    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:00:58.033143 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.031805    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:00:58.036179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.036154    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:00:58.038856 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:58.038809    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffd34de_62a6_4fac_9579_e0e735d2fddc.slice/crio-8f6a37e05dd1151dd9f841c7fc85b108498797de1cb5ae37f911f591d12b6e14 WatchSource:0}: Error finding container 8f6a37e05dd1151dd9f841c7fc85b108498797de1cb5ae37f911f591d12b6e14: Status 404 returned error can't find the container with id 8f6a37e05dd1151dd9f841c7fc85b108498797de1cb5ae37f911f591d12b6e14
Oct 30 04:00:58.041039 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.041022    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.045847 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.045798    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057430    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\""
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057498    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057542    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-kctfp\""
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057648    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\""
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057711    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\""
Oct 30 04:00:58.057995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.057430    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\""
Oct 30 04:00:58.063494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.063448    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.063929 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.063739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.064139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.063775    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.064358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.064305    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.064478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.064441    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:00:58.064655 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.064579    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.064789 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.064676    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.064789 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.064706    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hpt\" (UniqueName: \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.116003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.115273    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" event={"ID":"effd34de-62a6-4fac-9579-e0e735d2fddc","Type":"ContainerStarted","Data":"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"}
Oct 30 04:00:58.116003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.115325    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" event={"ID":"effd34de-62a6-4fac-9579-e0e735d2fddc","Type":"ContainerStarted","Data":"8f6a37e05dd1151dd9f841c7fc85b108498797de1cb5ae37f911f591d12b6e14"}
Oct 30 04:00:58.116003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.115946    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:00:58.154444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.154294    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" podStartSLOduration=1.154276275 podStartE2EDuration="1.154276275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:58.152953164 +0000 UTC m=+4816.377202444" watchObservedRunningTime="2026-10-30 04:00:58.154276275 +0000 UTC m=+4816.378525544"
Oct 30 04:00:58.165334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165304    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165477 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hpt\" (UniqueName: \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165477 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165585 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165614    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165687    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.165766 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.165733    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.166371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.166309    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.166371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.166312    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.166745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.166723    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.168153 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.168129    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.168153 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.168141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.168697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.168670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.186548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.186521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hpt\" (UniqueName: \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\") pod \"tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.198627 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.198598    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:00:58.202788 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:58.202759    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd0c23_8dce_445b_a46f_605945dd6258.slice/crio-27c1f9cae7d5626eb3f6be082a70669670ce1d9879524f62655892ebdf1dc807 WatchSource:0}: Error finding container 27c1f9cae7d5626eb3f6be082a70669670ce1d9879524f62655892ebdf1dc807: Status 404 returned error can't find the container with id 27c1f9cae7d5626eb3f6be082a70669670ce1d9879524f62655892ebdf1dc807
Oct 30 04:00:58.258227 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.258146    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:00:58.262564 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:58.262537    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12 WatchSource:0}: Error finding container 2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12: Status 404 returned error can't find the container with id 2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12
Oct 30 04:00:58.332220 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.332185    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:00:58.341057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.340643    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:00:58.345314 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:00:58.345286    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="f4d145b6-4071-4343-bac0-90aff3645652"
Oct 30 04:00:58.345782 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:58.345759    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d6d4d9_6bb6_42a2_9618_944eb2b367bc.slice/crio-ab116fcf846bac72caacd4102f8217005014eab12d34559cb2d9015627784220 WatchSource:0}: Error finding container ab116fcf846bac72caacd4102f8217005014eab12d34559cb2d9015627784220: Status 404 returned error can't find the container with id ab116fcf846bac72caacd4102f8217005014eab12d34559cb2d9015627784220
Oct 30 04:00:58.362574 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.362537    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:00:58.706127 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:58.706031    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:00:58.709646 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:00:58.709620    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb216b4ab_f130_4fc7_a7f0_5bb10a2bea18.slice/crio-a4b91f6e483bbc35fc4f805be8f1157e171b30f1fd9fe10d30395fae8d16d58c WatchSource:0}: Error finding container a4b91f6e483bbc35fc4f805be8f1157e171b30f1fd9fe10d30395fae8d16d58c: Status 404 returned error can't find the container with id a4b91f6e483bbc35fc4f805be8f1157e171b30f1fd9fe10d30395fae8d16d58c
Oct 30 04:00:59.121811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.121769    4528 generic.go:356] "Generic (PLEG): container finished" podID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerID="da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e" exitCode=1
Oct 30 04:00:59.122016 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.121893    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" event={"ID":"f29e2b19-ae28-4f33-a79d-af418b0a1052","Type":"ContainerDied","Data":"da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"}
Oct 30 04:00:59.122016 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.121978    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" event={"ID":"f29e2b19-ae28-4f33-a79d-af418b0a1052","Type":"ContainerStarted","Data":"2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12"}
Oct 30 04:00:59.122330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.122314    4528 scope.go:122] "RemoveContainer" containerID="da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"
Oct 30 04:00:59.124732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.124699    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" event={"ID":"28fd0c23-8dce-445b-a46f-605945dd6258","Type":"ContainerStarted","Data":"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"}
Oct 30 04:00:59.124867 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.124747    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" event={"ID":"28fd0c23-8dce-445b-a46f-605945dd6258","Type":"ContainerStarted","Data":"27c1f9cae7d5626eb3f6be082a70669670ce1d9879524f62655892ebdf1dc807"}
Oct 30 04:00:59.124867 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.124862    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:00:59.126425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.126403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerStarted","Data":"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"}
Oct 30 04:00:59.126501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.126454    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerStarted","Data":"a4b91f6e483bbc35fc4f805be8f1157e171b30f1fd9fe10d30395fae8d16d58c"}
Oct 30 04:00:59.128030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.128006    4528 generic.go:356] "Generic (PLEG): container finished" podID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerID="28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913" exitCode=1
Oct 30 04:00:59.128183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.128088    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" event={"ID":"39d6d4d9-6bb6-42a2-9618-944eb2b367bc","Type":"ContainerDied","Data":"28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"}
Oct 30 04:00:59.128183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.128139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" event={"ID":"39d6d4d9-6bb6-42a2-9618-944eb2b367bc","Type":"ContainerStarted","Data":"ab116fcf846bac72caacd4102f8217005014eab12d34559cb2d9015627784220"}
Oct 30 04:00:59.128275 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.128226    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:59.128275 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.128235    4528 scope.go:122] "RemoveContainer" containerID="28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"
Oct 30 04:00:59.137576 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.137454    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:00:59.172911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.172887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\") pod \"f4d145b6-4071-4343-bac0-90aff3645652\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") "
Oct 30 04:00:59.173042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.172933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\") pod \"f4d145b6-4071-4343-bac0-90aff3645652\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") "
Oct 30 04:00:59.173042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.172992    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\") pod \"f4d145b6-4071-4343-bac0-90aff3645652\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") "
Oct 30 04:00:59.173160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.173048    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\") pod \"f4d145b6-4071-4343-bac0-90aff3645652\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") "
Oct 30 04:00:59.173160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.173105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\" (UniqueName: \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\") pod \"f4d145b6-4071-4343-bac0-90aff3645652\" (UID: \"f4d145b6-4071-4343-bac0-90aff3645652\") "
Oct 30 04:00:59.173335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.173310    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle" pod "f4d145b6-4071-4343-bac0-90aff3645652" (UID: "f4d145b6-4071-4343-bac0-90aff3645652"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:59.173425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.173382    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf" pod "f4d145b6-4071-4343-bac0-90aff3645652" (UID: "f4d145b6-4071-4343-bac0-90aff3645652"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:59.173899 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.173877    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.174812 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.174793    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f4d145b6-4071-4343-bac0-90aff3645652-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.176305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.176284    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls" pod "f4d145b6-4071-4343-bac0-90aff3645652" (UID: "f4d145b6-4071-4343-bac0-90aff3645652"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:59.177029 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.177004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj" pod "f4d145b6-4071-4343-bac0-90aff3645652" (UID: "f4d145b6-4071-4343-bac0-90aff3645652"). InnerVolumeSpecName "kube-api-access-fn4jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:59.177585 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.177569    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key" pod "f4d145b6-4071-4343-bac0-90aff3645652" (UID: "f4d145b6-4071-4343-bac0-90aff3645652"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:59.275184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.275143    4528 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-storage-gcs-key\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.275184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.275179    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/f4d145b6-4071-4343-bac0-90aff3645652-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.275359 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.275194    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4jj\" (UniqueName: \"kubernetes.io/projected/f4d145b6-4071-4343-bac0-90aff3645652-kube-api-access-fn4jj\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:00:59.278287 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:00:59.278229    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" podStartSLOduration=2.278217522 podStartE2EDuration="2.278217522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:59.277305274 +0000 UTC m=+4817.501554551" watchObservedRunningTime="2026-10-30 04:00:59.278217522 +0000 UTC m=+4817.502466799"
Oct 30 04:01:00.136130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.136095    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerStarted","Data":"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"}
Oct 30 04:01:00.136581 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.136328    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:00.137628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.137599    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.89:8083/ready\": dial tcp 10.133.0.89:8083: connect: connection refused"
Oct 30 04:01:00.138198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.138168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" event={"ID":"39d6d4d9-6bb6-42a2-9618-944eb2b367bc","Type":"ContainerStarted","Data":"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"}
Oct 30 04:01:00.138306 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.138236    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:01:00.140021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.139998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" event={"ID":"f29e2b19-ae28-4f33-a79d-af418b0a1052","Type":"ContainerStarted","Data":"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"}
Oct 30 04:01:00.140106 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.140059    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.140290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.140273    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:01:00.178513 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.178408    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" podStartSLOduration=2.178395535 podStartE2EDuration="2.178395535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:00.177141515 +0000 UTC m=+4818.401390791" watchObservedRunningTime="2026-10-30 04:01:00.178395535 +0000 UTC m=+4818.402644812"
Oct 30 04:01:00.217204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.217150    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podStartSLOduration=3.217131595 podStartE2EDuration="3.217131595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:00.21566982 +0000 UTC m=+4818.439919096" watchObservedRunningTime="2026-10-30 04:01:00.217131595 +0000 UTC m=+4818.441380863"
Oct 30 04:01:00.255423 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.255368    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" podStartSLOduration=3.255353473 podStartE2EDuration="3.255353473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:00.254633851 +0000 UTC m=+4818.478883125" watchObservedRunningTime="2026-10-30 04:01:00.255353473 +0000 UTC m=+4818.479602750"
Oct 30 04:01:00.311193 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.311144    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:01:00.315523 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.315483    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:01:00.355923 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.355887    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:01:00.360806 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.360779    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.374008 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.373986    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\""
Oct 30 04:01:00.381326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.381289    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:01:00.484886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.484772    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.485026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.484897    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.485026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.484927    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcjh\" (UniqueName: \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.485093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.485025    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.485093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.485042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.585977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.585938    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.585977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.585979    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcjh\" (UniqueName: \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.586178 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.586026    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.586178 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.586043    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.586178 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.586093    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.586807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.586779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.586945 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.586914    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.588508 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.588490    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.588569 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.588550    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:00.607900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:00.607868    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcjh\" (UniqueName: \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:01.145667 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:01.145616    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.89:8083/ready\": dial tcp 10.133.0.89:8083: connect: connection refused"
Oct 30 04:01:01.146131 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:01.145721    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:01.232233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:01.232206    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:01.892714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:01.892683    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d145b6-4071-4343-bac0-90aff3645652" path="/var/lib/kubelet/pods/f4d145b6-4071-4343-bac0-90aff3645652/volumes"
Oct 30 04:01:02.151433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:02.151354    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:02.747703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:02.747666    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:01:04.159510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:04.159468    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway" containerID="cri-o://e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2" gracePeriod=30
Oct 30 04:01:04.159510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:04.159501    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa" containerID="cri-o://3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa" gracePeriod=30
Oct 30 04:01:04.631740 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:04.631706    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:01:05.166200 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.166171    4528 generic.go:356] "Generic (PLEG): container finished" podID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerID="3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa" exitCode=0
Oct 30 04:01:05.166590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.166246    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerDied","Data":"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"}
Oct 30 04:01:05.508551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.508523    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:05.633366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633403    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633432    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633463    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633505    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\" (UniqueName: \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.633813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.633636    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\") pod \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\" (UID: \"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18\") "
Oct 30 04:01:05.634110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.634080    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:05.634110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.634106    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:05.634476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.634389    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:05.635617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.635593    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:05.635795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.635776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "kube-api-access-57hpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:05.635944 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.635903    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:05.640767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.640747    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant" pod "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" (UID: "b216b4ab-f130-4fc7-a7f0-5bb10a2bea18"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734417    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734444    4528 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734455    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734464    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57hpt\" (UniqueName: \"kubernetes.io/projected/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-kube-api-access-57hpt\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734473    4528 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734480    4528 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-tenant\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:05.734490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:05.734488    4528 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18-serving-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:06.171728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.171691    4528 generic.go:356] "Generic (PLEG): container finished" podID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerID="e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2" exitCode=0
Oct 30 04:01:06.172198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.171753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerDied","Data":"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"}
Oct 30 04:01:06.172198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.171813    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj" event={"ID":"b216b4ab-f130-4fc7-a7f0-5bb10a2bea18","Type":"ContainerDied","Data":"a4b91f6e483bbc35fc4f805be8f1157e171b30f1fd9fe10d30395fae8d16d58c"}
Oct 30 04:01:06.172198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.171758    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"
Oct 30 04:01:06.172198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.171868    4528 scope.go:122] "RemoveContainer" containerID="3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"
Oct 30 04:01:06.180110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.180092    4528 scope.go:122] "RemoveContainer" containerID="e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"
Oct 30 04:01:06.187772 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.187756    4528 scope.go:122] "RemoveContainer" containerID="3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"
Oct 30 04:01:06.188015 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:06.187998    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa\": container with ID starting with 3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa not found: ID does not exist" containerID="3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"
Oct 30 04:01:06.188066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.188023    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa"} err="failed to get container status \"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa\": rpc error: code = NotFound desc = could not find container \"3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa\": container with ID starting with 3d9e0668124921d56a927adcf96242d293f12cbb3d3906f5560ae01fbe966aaa not found: ID does not exist"
Oct 30 04:01:06.188066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.188039    4528 scope.go:122] "RemoveContainer" containerID="e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"
Oct 30 04:01:06.188255 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:06.188235    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2\": container with ID starting with e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2 not found: ID does not exist" containerID="e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"
Oct 30 04:01:06.188313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.188264    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2"} err="failed to get container status \"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2\": rpc error: code = NotFound desc = could not find container \"e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2\": container with ID starting with e0e57ced677d87426138d4d2f97134dc68c8e26e124ff9ad17ee5ba69494fdb2 not found: ID does not exist"
Oct 30 04:01:06.217860 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.217837    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:01:06.228307 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:06.228285    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-9948b5b9b-8f4dj"]
Oct 30 04:01:07.145309 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.145277    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.170657 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.170624    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.246034 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.246001    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.246443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.246164    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.258552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.258527    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") DevicePath \"csi-7aa53e340b2b8925302958b156c025cf68a27065b5ecfbbdd417e244edeb1c66\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.358876 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.358841    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7e9a666e0ba128a61b3fe9d2d9fd641d359a39f007a46310717d51259c1b73d9/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.382725 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.382695    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.573051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.572960    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:07.881282 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.881244    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:01:07.885527 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:01:07.885502    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6b297c_4414_4511_bc49_cbdefc2c0c5a.slice/crio-128a2e1fb01d2b909dfcf39c08166a0d93de831ee4013b980ec511ac074b88c9 WatchSource:0}: Error finding container 128a2e1fb01d2b909dfcf39c08166a0d93de831ee4013b980ec511ac074b88c9: Status 404 returned error can't find the container with id 128a2e1fb01d2b909dfcf39c08166a0d93de831ee4013b980ec511ac074b88c9
Oct 30 04:01:07.892463 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:07.892440    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" path="/var/lib/kubelet/pods/b216b4ab-f130-4fc7-a7f0-5bb10a2bea18/volumes"
Oct 30 04:01:08.182911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:08.182810    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7c6b297c-4414-4511-bc49-cbdefc2c0c5a","Type":"ContainerStarted","Data":"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"}
Oct 30 04:01:08.182911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:08.182885    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7c6b297c-4414-4511-bc49-cbdefc2c0c5a","Type":"ContainerStarted","Data":"128a2e1fb01d2b909dfcf39c08166a0d93de831ee4013b980ec511ac074b88c9"}
Oct 30 04:01:08.183098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:08.182934    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:08.222357 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:08.222300    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=8.222286061 podStartE2EDuration="8.222286061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:08.221081748 +0000 UTC m=+4826.445331035" watchObservedRunningTime="2026-10-30 04:01:08.222286061 +0000 UTC m=+4826.446535337"
Oct 30 04:01:20.145446 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:20.145417    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:01:20.230107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:20.230079    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:01:21.150991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:21.150960    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:01:21.151690 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:21.151660    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:23.669991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:23.669953    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:01:23.670492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:23.670386    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo" containerID="cri-o://d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587" gracePeriod=30
Oct 30 04:01:23.717029 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:23.716997    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:01:23.717217 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:23.717198    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" podUID="effd34de-62a6-4fac-9579-e0e735d2fddc" containerName="tempo" containerID="cri-o://fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee" gracePeriod=30
Oct 30 04:01:24.474620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.474593    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:01:24.601390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601297    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\" (UniqueName: \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\") pod \"effd34de-62a6-4fac-9579-e0e735d2fddc\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") "
Oct 30 04:01:24.601390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601347    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\") pod \"effd34de-62a6-4fac-9579-e0e735d2fddc\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") "
Oct 30 04:01:24.601390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601375    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\") pod \"effd34de-62a6-4fac-9579-e0e735d2fddc\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") "
Oct 30 04:01:24.601694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601407    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\") pod \"effd34de-62a6-4fac-9579-e0e735d2fddc\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") "
Oct 30 04:01:24.601694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601464    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\") pod \"effd34de-62a6-4fac-9579-e0e735d2fddc\" (UID: \"effd34de-62a6-4fac-9579-e0e735d2fddc\") "
Oct 30 04:01:24.601814 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage" pod "effd34de-62a6-4fac-9579-e0e735d2fddc" (UID: "effd34de-62a6-4fac-9579-e0e735d2fddc"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:24.601928 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601898    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle" pod "effd34de-62a6-4fac-9579-e0e735d2fddc" (UID: "effd34de-62a6-4fac-9579-e0e735d2fddc"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:24.601928 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.601919    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf" pod "effd34de-62a6-4fac-9579-e0e735d2fddc" (UID: "effd34de-62a6-4fac-9579-e0e735d2fddc"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:24.603619 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.603599    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls" pod "effd34de-62a6-4fac-9579-e0e735d2fddc" (UID: "effd34de-62a6-4fac-9579-e0e735d2fddc"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:24.603708 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.603691    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77" pod "effd34de-62a6-4fac-9579-e0e735d2fddc" (UID: "effd34de-62a6-4fac-9579-e0e735d2fddc"). InnerVolumeSpecName "kube-api-access-2cc77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:24.699712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.699642    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:01:24.700286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.700229    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo" containerID="cri-o://8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295" gracePeriod=30
Oct 30 04:01:24.702238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.702210    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cc77\" (UniqueName: \"kubernetes.io/projected/effd34de-62a6-4fac-9579-e0e735d2fddc-kube-api-access-2cc77\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:24.702305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.702246    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:24.702305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.702263    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:24.702305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.702276    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:24.702305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:24.702290    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/effd34de-62a6-4fac-9579-e0e735d2fddc-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:25.262940 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.262902    4528 generic.go:356] "Generic (PLEG): container finished" podID="effd34de-62a6-4fac-9579-e0e735d2fddc" containerID="fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee" exitCode=0
Oct 30 04:01:25.263122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.262953    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"
Oct 30 04:01:25.263122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.262955    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" event={"ID":"effd34de-62a6-4fac-9579-e0e735d2fddc","Type":"ContainerDied","Data":"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"}
Oct 30 04:01:25.263122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.263009    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw" event={"ID":"effd34de-62a6-4fac-9579-e0e735d2fddc","Type":"ContainerDied","Data":"8f6a37e05dd1151dd9f841c7fc85b108498797de1cb5ae37f911f591d12b6e14"}
Oct 30 04:01:25.263122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.263039    4528 scope.go:122] "RemoveContainer" containerID="fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"
Oct 30 04:01:25.271693 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.271668    4528 scope.go:122] "RemoveContainer" containerID="fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"
Oct 30 04:01:25.271991 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:25.271971    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee\": container with ID starting with fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee not found: ID does not exist" containerID="fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"
Oct 30 04:01:25.272049 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.272000    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee"} err="failed to get container status \"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee\": rpc error: code = NotFound desc = could not find container \"fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee\": container with ID starting with fb7c49f6b8b27ce9f91ef140517782c9fcf5e1b6d2b1441fd3986d627feeabee not found: ID does not exist"
Oct 30 04:01:25.308036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.308004    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:01:25.323116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.323081    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-5f59c684f-9fxbw"]
Oct 30 04:01:25.891807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:25.891778    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effd34de-62a6-4fac-9579-e0e735d2fddc" path="/var/lib/kubelet/pods/effd34de-62a6-4fac-9579-e0e735d2fddc/volumes"
Oct 30 04:01:29.192836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:29.192790    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:30.529738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:30.529695    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:31.149981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:31.149937    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:31.150706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:31.150673    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:34.631773 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:34.631736    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:01:39.191229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:39.191192    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:39.613448 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:39.613359    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:40.430272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:40.430228    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:41.149506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:41.149469    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:41.149693 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:41.149481    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:41.943728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:41.943689    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:49.191273 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:49.191240    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:01:50.330631 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:50.330577    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:50.331128 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:50.330730    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:01:51.149656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:51.149618    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:51.149871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:51.149728    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:01:51.150129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:51.150104    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:01:54.318431 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.318403    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:01:54.389840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.389790    4528 generic.go:356] "Generic (PLEG): container finished" podID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerID="d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587" exitCode=137
Oct 30 04:01:54.390036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.389860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"
Oct 30 04:01:54.390036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.389865    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" event={"ID":"f29e2b19-ae28-4f33-a79d-af418b0a1052","Type":"ContainerDied","Data":"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"}
Oct 30 04:01:54.390036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.389937    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr" event={"ID":"f29e2b19-ae28-4f33-a79d-af418b0a1052","Type":"ContainerDied","Data":"2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12"}
Oct 30 04:01:54.390036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.389973    4528 scope.go:122] "RemoveContainer" containerID="d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"
Oct 30 04:01:54.398389 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.398370    4528 scope.go:122] "RemoveContainer" containerID="da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"
Oct 30 04:01:54.405993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.405976    4528 scope.go:122] "RemoveContainer" containerID="d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"
Oct 30 04:01:54.406253 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:54.406234    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587\": container with ID starting with d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587 not found: ID does not exist" containerID="d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"
Oct 30 04:01:54.406324 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.406262    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587"} err="failed to get container status \"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587\": rpc error: code = NotFound desc = could not find container \"d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587\": container with ID starting with d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587 not found: ID does not exist"
Oct 30 04:01:54.406324 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.406279    4528 scope.go:122] "RemoveContainer" containerID="da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"
Oct 30 04:01:54.406524 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:54.406505    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e\": container with ID starting with da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e not found: ID does not exist" containerID="da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"
Oct 30 04:01:54.406591 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.406530    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e"} err="failed to get container status \"da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e\": rpc error: code = NotFound desc = could not find container \"da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e\": container with ID starting with da57d22803dd48b9c7e57251f99f277065ed21bea3f8b59809ebba8c8386743e not found: ID does not exist"
Oct 30 04:01:54.447975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.447901    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.447975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.447947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.447975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.447974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\" (UniqueName: \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.448204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.447997    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.448204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.448030    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.448204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.448061    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\") pod \"f29e2b19-ae28-4f33-a79d-af418b0a1052\" (UID: \"f29e2b19-ae28-4f33-a79d-af418b0a1052\") "
Oct 30 04:01:54.448360 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.448335    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:54.448535 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.448508    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:54.448634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.448511    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:54.450270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.450241    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "kube-api-access-bklsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:54.450270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.450251    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:54.450402 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.450310    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls" pod "f29e2b19-ae28-4f33-a79d-af418b0a1052" (UID: "f29e2b19-ae28-4f33-a79d-af418b0a1052"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:54.548967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.548925    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.548967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.548959    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.548967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.548971    4528 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/f29e2b19-ae28-4f33-a79d-af418b0a1052-storage-gcs-key\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.549206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.548982    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.549206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.548991    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklsk\" (UniqueName: \"kubernetes.io/projected/f29e2b19-ae28-4f33-a79d-af418b0a1052-kube-api-access-bklsk\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.549206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.549000    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/f29e2b19-ae28-4f33-a79d-af418b0a1052-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:54.732151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.731713    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:01:54.742716 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.742679    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-65746454dc-gnhfr"]
Oct 30 04:01:54.753391 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:54.753356    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd0c23_8dce_445b_a46f_605945dd6258.slice/crio-conmon-8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 04:01:54.753529 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:54.753384    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-conmon-d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd0c23_8dce_445b_a46f_605945dd6258.slice/crio-8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fd0c23_8dce_445b_a46f_605945dd6258.slice/crio-conmon-8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-d12ad5c9c7d5791cbca35157e4058e733737c72743330c8e7df8218752d58587.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 04:01:54.881999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:54.881974    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:01:55.053353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053257    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053311    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053387    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\" (UniqueName: \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053413    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053443    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\") pod \"28fd0c23-8dce-445b-a46f-605945dd6258\" (UID: \"28fd0c23-8dce-445b-a46f-605945dd6258\") "
Oct 30 04:01:55.053886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053859    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:55.053956 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:55.054015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.053937    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:55.055614 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.055583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:55.055614 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.055596    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:55.055771 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.055678    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z" pod "28fd0c23-8dce-445b-a46f-605945dd6258" (UID: "28fd0c23-8dce-445b-a46f-605945dd6258"). InnerVolumeSpecName "kube-api-access-5k96z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:55.154489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154452    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k96z\" (UniqueName: \"kubernetes.io/projected/28fd0c23-8dce-445b-a46f-605945dd6258-kube-api-access-5k96z\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.154489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154483    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.154489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154496    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.154728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154506    4528 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-storage-gcs-key\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.154728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154517    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.154728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.154525    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/28fd0c23-8dce-445b-a46f-605945dd6258-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:01:55.396548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.396513    4528 generic.go:356] "Generic (PLEG): container finished" podID="28fd0c23-8dce-445b-a46f-605945dd6258" containerID="8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295" exitCode=137
Oct 30 04:01:55.396969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.396560    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"
Oct 30 04:01:55.396969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.396566    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" event={"ID":"28fd0c23-8dce-445b-a46f-605945dd6258","Type":"ContainerDied","Data":"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"}
Oct 30 04:01:55.396969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.396606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq" event={"ID":"28fd0c23-8dce-445b-a46f-605945dd6258","Type":"ContainerDied","Data":"27c1f9cae7d5626eb3f6be082a70669670ce1d9879524f62655892ebdf1dc807"}
Oct 30 04:01:55.396969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.396627    4528 scope.go:122] "RemoveContainer" containerID="8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"
Oct 30 04:01:55.405952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.405933    4528 scope.go:122] "RemoveContainer" containerID="8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"
Oct 30 04:01:55.406239 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:55.406222    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295\": container with ID starting with 8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295 not found: ID does not exist" containerID="8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"
Oct 30 04:01:55.406277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.406249    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295"} err="failed to get container status \"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295\": rpc error: code = NotFound desc = could not find container \"8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295\": container with ID starting with 8533dadfb4ec0e9590665a719d1cdb8c93858bc47502978b42ee41d1f27b7295 not found: ID does not exist"
Oct 30 04:01:55.444875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.444842    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:01:55.454502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.454466    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-b78f9dcd8-4vbwq"]
Oct 30 04:01:55.892416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.892382    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" path="/var/lib/kubelet/pods/28fd0c23-8dce-445b-a46f-605945dd6258/volumes"
Oct 30 04:01:55.892811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:55.892796    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" path="/var/lib/kubelet/pods/f29e2b19-ae28-4f33-a79d-af418b0a1052/volumes"
Oct 30 04:01:59.003916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:59.003882    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:59.116776 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:01:59.116742    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:01:59.305308 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:01:59.305211    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:01.149805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:01.149759    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:02:04.630890 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:04.630855    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:02:04.797138 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:04.797105    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:05.681706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:05.681668    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:02:05.682185 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:05.681934    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" containerID="cri-o://c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181" gracePeriod=30
Oct 30 04:02:05.685975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:05.685949    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:02:07.837940 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.837914    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:02:07.874618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874581    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874633    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874668    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874725    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\" (UniqueName: \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874871    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874942    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\") pod \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\" (UID: \"39d6d4d9-6bb6-42a2-9618-944eb2b367bc\") "
Oct 30 04:02:07.874996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.874989    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:02:07.875195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.875161    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:07.875250 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.875180    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-tmp-storage\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:07.875295 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.875279    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:07.878868 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.877257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:07.878868 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.877338    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:07.878868 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.877734    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr" pod "39d6d4d9-6bb6-42a2-9618-944eb2b367bc" (UID: "39d6d4d9-6bb6-42a2-9618-944eb2b367bc"). InnerVolumeSpecName "kube-api-access-5bmkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:07.975628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.975592    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:07.975628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.975623    4528 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-storage-gcs-key\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:07.975910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.975637    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmkr\" (UniqueName: \"kubernetes.io/projected/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-kube-api-access-5bmkr\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:07.975910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.975651    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:07.975910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:07.975665    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/39d6d4d9-6bb6-42a2-9618-944eb2b367bc-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:08.452981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.452940    4528 generic.go:356] "Generic (PLEG): container finished" podID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerID="c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181" exitCode=0
Oct 30 04:02:08.453165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.453015    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"
Oct 30 04:02:08.453165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.453015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" event={"ID":"39d6d4d9-6bb6-42a2-9618-944eb2b367bc","Type":"ContainerDied","Data":"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"}
Oct 30 04:02:08.453165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.453072    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp" event={"ID":"39d6d4d9-6bb6-42a2-9618-944eb2b367bc","Type":"ContainerDied","Data":"ab116fcf846bac72caacd4102f8217005014eab12d34559cb2d9015627784220"}
Oct 30 04:02:08.453165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.453101    4528 scope.go:122] "RemoveContainer" containerID="c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"
Oct 30 04:02:08.463467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.463435    4528 scope.go:122] "RemoveContainer" containerID="28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"
Oct 30 04:02:08.470997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.470975    4528 scope.go:122] "RemoveContainer" containerID="c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"
Oct 30 04:02:08.471280 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:08.471256    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181\": container with ID starting with c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181 not found: ID does not exist" containerID="c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"
Oct 30 04:02:08.471357 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.471287    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181"} err="failed to get container status \"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181\": rpc error: code = NotFound desc = could not find container \"c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181\": container with ID starting with c9cf2d2ee3bd57ddcf0a8b0a572980a6dd691e8961860877782ef19d4d234181 not found: ID does not exist"
Oct 30 04:02:08.471357 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.471304    4528 scope.go:122] "RemoveContainer" containerID="28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"
Oct 30 04:02:08.471553 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:08.471533    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913\": container with ID starting with 28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913 not found: ID does not exist" containerID="28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"
Oct 30 04:02:08.471597 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.471561    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913"} err="failed to get container status \"28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913\": rpc error: code = NotFound desc = could not find container \"28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913\": container with ID starting with 28a740de36a361f95b5d6ff0dc481968a9365d06f107c89ed9097ffef1a29913 not found: ID does not exist"
Oct 30 04:02:08.498015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.497994    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:02:08.508597 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.508575    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-759575fd59-52wzp"]
Oct 30 04:02:08.784620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.784531    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:08.898924 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:08.898890    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:09.016486 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.016444    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:09.387203 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.387058    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:09.387456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.387404    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1327d07c-a646-455f-9f6f-05ff43c46b3a" containerName="otc-container" containerID="cri-o://e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b" gracePeriod=30
Oct 30 04:02:09.536064 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.536031    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:09.624594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.624569    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:09.649531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.649462    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:09.689740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.689703    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\") pod \"1327d07c-a646-455f-9f6f-05ff43c46b3a\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") "
Oct 30 04:02:09.689895 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.689749    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\") pod \"1327d07c-a646-455f-9f6f-05ff43c46b3a\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") "
Oct 30 04:02:09.689895 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.689878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\" (UniqueName: \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\") pod \"1327d07c-a646-455f-9f6f-05ff43c46b3a\" (UID: \"1327d07c-a646-455f-9f6f-05ff43c46b3a\") "
Oct 30 04:02:09.690098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.690077    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal" pod "1327d07c-a646-455f-9f6f-05ff43c46b3a" (UID: "1327d07c-a646-455f-9f6f-05ff43c46b3a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:09.690166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.690152    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1327d07c-a646-455f-9f6f-05ff43c46b3a-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:09.691861 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.691844    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs" pod "1327d07c-a646-455f-9f6f-05ff43c46b3a" (UID: "1327d07c-a646-455f-9f6f-05ff43c46b3a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:09.691969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.691902    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79" pod "1327d07c-a646-455f-9f6f-05ff43c46b3a" (UID: "1327d07c-a646-455f-9f6f-05ff43c46b3a"). InnerVolumeSpecName "kube-api-access-t8c79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:09.790679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.790645    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8c79\" (UniqueName: \"kubernetes.io/projected/1327d07c-a646-455f-9f6f-05ff43c46b3a-kube-api-access-t8c79\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:09.790679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.790674    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1327d07c-a646-455f-9f6f-05ff43c46b3a-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:09.892101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:09.892075    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" path="/var/lib/kubelet/pods/39d6d4d9-6bb6-42a2-9618-944eb2b367bc/volumes"
Oct 30 04:02:10.463872 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.463811    4528 generic.go:356] "Generic (PLEG): container finished" podID="1327d07c-a646-455f-9f6f-05ff43c46b3a" containerID="e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b" exitCode=0
Oct 30 04:02:10.464403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.463874    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1327d07c-a646-455f-9f6f-05ff43c46b3a","Type":"ContainerDied","Data":"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"}
Oct 30 04:02:10.464403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.463915    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1327d07c-a646-455f-9f6f-05ff43c46b3a","Type":"ContainerDied","Data":"08b0263a2afed035f9a7bf09c2b69c7d053fc53ae10f9d6aff2a76e6fb6b3f2a"}
Oct 30 04:02:10.464403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.463937    4528 scope.go:122] "RemoveContainer" containerID="e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"
Oct 30 04:02:10.464403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.463880    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.471862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.471839    4528 scope.go:122] "RemoveContainer" containerID="e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"
Oct 30 04:02:10.472148 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:10.472128    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b\": container with ID starting with e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b not found: ID does not exist" containerID="e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"
Oct 30 04:02:10.472222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.472158    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b"} err="failed to get container status \"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b\": rpc error: code = NotFound desc = could not find container \"e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b\": container with ID starting with e3b0ab4aba3603f13d1cb477f03c2e1557795553c209d0e21a03c1dde4e15d2b not found: ID does not exist"
Oct 30 04:02:10.508757 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.508725    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:10.519283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.519255    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:10.551291 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551258    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:10.551775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551752    4528 cpu_manager.go:414] "containerMap: removing container" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:10.551775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551772    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551783    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551788    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551798    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551804    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551828    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551836    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551852    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1327d07c-a646-455f-9f6f-05ff43c46b3a" containerName="otc-container"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551857    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327d07c-a646-455f-9f6f-05ff43c46b3a" containerName="otc-container"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551864    4528 cpu_manager.go:414] "containerMap: removing container" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551868    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551877    4528 cpu_manager.go:414] "containerMap: removing container" podUID="effd34de-62a6-4fac-9579-e0e735d2fddc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551881    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effd34de-62a6-4fac-9579-e0e735d2fddc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551932    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fd0c23-8dce-445b-a46f-605945dd6258" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551942    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="effd34de-62a6-4fac-9579-e0e735d2fddc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551948    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551956    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551961    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:10.551960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551968    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway"
Oct 30 04:02:10.552682 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551974    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b216b4ab-f130-4fc7-a7f0-5bb10a2bea18" containerName="tempo-gateway-opa"
Oct 30 04:02:10.552682 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.551980    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327d07c-a646-455f-9f6f-05ff43c46b3a" containerName="otc-container"
Oct 30 04:02:10.556184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.556169    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.569321 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.569301    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:02:10.569453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.569436    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-vv788\""
Oct 30 04:02:10.569509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.569441    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-90b17333\""
Oct 30 04:02:10.575637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.575558    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:10.596010 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.595980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.596110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.596034    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.596177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.596108    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfg9\" (UniqueName: \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.621358 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:10.621331    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:10.621473 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:10.621392    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:10.697012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.696969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.697204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.697019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.697204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.697047    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfg9\" (UniqueName: \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.697711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.697688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.699446 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.699422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.719164 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.719105    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfg9\" (UniqueName: \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\") pod \"data-science-collector-collector-1\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:10.868308 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:10.868274    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:11.163154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.163092    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:11.168365 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:02:11.168338    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af4b389_3307_4a10_9174_bba10632cf00.slice/crio-278c9fd6abec0b8dd81392be2d93b17e1f96a4bbbb7257dd1c51325dfa7f3252 WatchSource:0}: Error finding container 278c9fd6abec0b8dd81392be2d93b17e1f96a4bbbb7257dd1c51325dfa7f3252: Status 404 returned error can't find the container with id 278c9fd6abec0b8dd81392be2d93b17e1f96a4bbbb7257dd1c51325dfa7f3252
Oct 30 04:02:11.172862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.171415    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:02:11.172862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.171558    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:02:11.470996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.470913    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4af4b389-3307-4a10-9174-bba10632cf00","Type":"ContainerStarted","Data":"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"}
Oct 30 04:02:11.470996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.470987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4af4b389-3307-4a10-9174-bba10632cf00","Type":"ContainerStarted","Data":"278c9fd6abec0b8dd81392be2d93b17e1f96a4bbbb7257dd1c51325dfa7f3252"}
Oct 30 04:02:11.509301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.509245    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.509231698 podStartE2EDuration="1.509231698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:11.508282076 +0000 UTC m=+4889.732531354" watchObservedRunningTime="2026-10-30 04:02:11.509231698 +0000 UTC m=+4889.733480972"
Oct 30 04:02:11.891927 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:11.891896    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1327d07c-a646-455f-9f6f-05ff43c46b3a" path="/var/lib/kubelet/pods/1327d07c-a646-455f-9f6f-05ff43c46b3a/volumes"
Oct 30 04:02:14.310618 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:14.310586    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:14.806907 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:14.806874    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:24.844830 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:24.844743    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:29.307758 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:29.307716    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:34.631553 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:34.631509    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:02:34.882258 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:34.882161    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e2b19_ae28_4f33_a79d_af418b0a1052.slice/crio-2dd7477d90fa657e65f5bda29c4a9e89e57a641f9beda8acb0576a1c3f788c12\": RecentStats: unable to find data in memory cache]"
Oct 30 04:02:39.746481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:39.746440    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:42.868026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:42.867937    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:42.868505 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:42.868271    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="4af4b389-3307-4a10-9174-bba10632cf00" containerName="otc-container" containerID="cri-o://73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8" gracePeriod=30
Oct 30 04:02:42.931072 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:42.931013    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:43.059098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.059071    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:43.127569 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.127512    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:43.184328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.184296    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:43.269865 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.269836    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\" (UniqueName: \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\") pod \"4af4b389-3307-4a10-9174-bba10632cf00\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") "
Oct 30 04:02:43.270024 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.269876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\") pod \"4af4b389-3307-4a10-9174-bba10632cf00\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") "
Oct 30 04:02:43.270024 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.269984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\") pod \"4af4b389-3307-4a10-9174-bba10632cf00\" (UID: \"4af4b389-3307-4a10-9174-bba10632cf00\") "
Oct 30 04:02:43.270345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.270324    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal" pod "4af4b389-3307-4a10-9174-bba10632cf00" (UID: "4af4b389-3307-4a10-9174-bba10632cf00"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:43.271876 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.271845    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs" pod "4af4b389-3307-4a10-9174-bba10632cf00" (UID: "4af4b389-3307-4a10-9174-bba10632cf00"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:43.272000 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.271930    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9" pod "4af4b389-3307-4a10-9174-bba10632cf00" (UID: "4af4b389-3307-4a10-9174-bba10632cf00"). InnerVolumeSpecName "kube-api-access-9kfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:43.370763 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.370731    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4af4b389-3307-4a10-9174-bba10632cf00-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:43.370763 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.370762    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfg9\" (UniqueName: \"kubernetes.io/projected/4af4b389-3307-4a10-9174-bba10632cf00-kube-api-access-9kfg9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:43.370964 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.370776    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4af4b389-3307-4a10-9174-bba10632cf00-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:43.448065 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.448029    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:43.467232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.467201    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:02:43.467468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.467442    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo" containerID="cri-o://5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901" gracePeriod=30
Oct 30 04:02:43.574317 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.574285    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:43.602599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.602566    4528 generic.go:356] "Generic (PLEG): container finished" podID="4af4b389-3307-4a10-9174-bba10632cf00" containerID="73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8" exitCode=0
Oct 30 04:02:43.602780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.602614    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:43.602780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.602643    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4af4b389-3307-4a10-9174-bba10632cf00","Type":"ContainerDied","Data":"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"}
Oct 30 04:02:43.602780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.602694    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"4af4b389-3307-4a10-9174-bba10632cf00","Type":"ContainerDied","Data":"278c9fd6abec0b8dd81392be2d93b17e1f96a4bbbb7257dd1c51325dfa7f3252"}
Oct 30 04:02:43.602780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.602716    4528 scope.go:122] "RemoveContainer" containerID="73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"
Oct 30 04:02:43.611977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.611954    4528 scope.go:122] "RemoveContainer" containerID="73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"
Oct 30 04:02:43.612282 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:43.612254    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8\": container with ID starting with 73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8 not found: ID does not exist" containerID="73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"
Oct 30 04:02:43.612394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.612288    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8"} err="failed to get container status \"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8\": rpc error: code = NotFound desc = could not find container \"73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8\": container with ID starting with 73e5a6b66d82659c45e5f1de57d1f739e35ebdf83de0b1641be1572fe77ef6a8 not found: ID does not exist"
Oct 30 04:02:43.648683 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.648651    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:43.652535 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.652510    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:43.892326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:43.892287    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af4b389-3307-4a10-9174-bba10632cf00" path="/var/lib/kubelet/pods/4af4b389-3307-4a10-9174-bba10632cf00/volumes"
Oct 30 04:02:44.041879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:44.041848    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"]
Oct 30 04:02:44.046109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:44.046086    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/fake-gcs-bucket-creator"]
Oct 30 04:02:45.892314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:45.892282    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f412fc54-b812-44a7-8d20-fbd6c87b4cf5" path="/var/lib/kubelet/pods/f412fc54-b812-44a7-8d20-fbd6c87b4cf5/volumes"
Oct 30 04:02:46.218259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:46.218148    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:47.216582 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.216559    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:02:47.299938 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.299904    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\" (UniqueName: \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.300114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.299960    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.300114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.299988    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.300114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.300007    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.300257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.300112    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.300452 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.300425    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:47.302185 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.302157    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "kube-api-access-bxcjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:02:47.302320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.302223    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:47.302320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.302307    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:02:47.302797 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.302778    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:02:47.400620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400583    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\") pod \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\" (UID: \"7c6b297c-4414-4511-bc49-cbdefc2c0c5a\") "
Oct 30 04:02:47.400813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400795    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:47.400880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400846    4528 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-storage-gcs-key\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:47.400880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400862    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:47.400967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400896    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:02:47.400967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400914    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcjh\" (UniqueName: \"kubernetes.io/projected/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-kube-api-access-bxcjh\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:47.401043 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.400989    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle" pod "7c6b297c-4414-4511-bc49-cbdefc2c0c5a" (UID: "7c6b297c-4414-4511-bc49-cbdefc2c0c5a"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:02:47.409159 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.409137    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:02:47.469919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.469883    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:47.501480 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.501447    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-7769207b-70b4-44ac-8fc1-0bb4abdb2bdc\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0445f74e44662c879\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-7aa53e340b2b8925302958b156c025cf68a27065b5ecfbbdd417e244edeb1c66\""
Oct 30 04:02:47.501480 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.501479    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b297c-4414-4511-bc49-cbdefc2c0c5a-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:02:47.585542 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.585500    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:47.619758 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.619727    4528 generic.go:356] "Generic (PLEG): container finished" podID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerID="5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901" exitCode=0
Oct 30 04:02:47.619968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.619767    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"
Oct 30 04:02:47.619968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.619780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7c6b297c-4414-4511-bc49-cbdefc2c0c5a","Type":"ContainerDied","Data":"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"}
Oct 30 04:02:47.619968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.619861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"7c6b297c-4414-4511-bc49-cbdefc2c0c5a","Type":"ContainerDied","Data":"128a2e1fb01d2b909dfcf39c08166a0d93de831ee4013b980ec511ac074b88c9"}
Oct 30 04:02:47.619968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.619891    4528 scope.go:122] "RemoveContainer" containerID="5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"
Oct 30 04:02:47.628445 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.628429    4528 scope.go:122] "RemoveContainer" containerID="5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"
Oct 30 04:02:47.628710 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:02:47.628687    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901\": container with ID starting with 5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901 not found: ID does not exist" containerID="5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"
Oct 30 04:02:47.628802 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.628713    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901"} err="failed to get container status \"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901\": rpc error: code = NotFound desc = could not find container \"5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901\": container with ID starting with 5e9c229d3b1500a595e0a1430ae8ea860e10f3750d328c9c8519cacd48740901 not found: ID does not exist"
Oct 30 04:02:47.667947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.667861    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:02:47.671904 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.671876    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"]
Oct 30 04:02:47.707637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.707466    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:47.896912 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:47.896877    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" path="/var/lib/kubelet/pods/7c6b297c-4414-4511-bc49-cbdefc2c0c5a/volumes"
Oct 30 04:02:48.153480 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.153451    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:48.153754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.153736    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo"
Oct 30 04:02:48.154125 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154097    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154146    4528 cpu_manager.go:414] "containerMap: removing container" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154156    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d6d4d9-6bb6-42a2-9618-944eb2b367bc" containerName="tempo"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154169    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154175    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154186    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4af4b389-3307-4a10-9174-bba10632cf00" containerName="otc-container"
Oct 30 04:02:48.154257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154196    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af4b389-3307-4a10-9174-bba10632cf00" containerName="otc-container"
Oct 30 04:02:48.154483 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154298    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af4b389-3307-4a10-9174-bba10632cf00" containerName="otc-container"
Oct 30 04:02:48.154483 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154314    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29e2b19-ae28-4f33-a79d-af418b0a1052" containerName="tempo"
Oct 30 04:02:48.154483 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.154324    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6b297c-4414-4511-bc49-cbdefc2c0c5a" containerName="tempo"
Oct 30 04:02:48.158916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.158891    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.178708 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.178680    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:48.186948 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.186925    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lsxgg\""
Oct 30 04:02:48.187096 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.186929    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:02:48.207091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.207055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.207362 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.207328    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6zf7\" (UniqueName: \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.207469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.207421    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.215579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.215551    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:48.255492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.255463    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:02:48.255492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.255492    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:02:48.255911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.255649    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-645168e1\""
Oct 30 04:02:48.308520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.308479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.308520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.308524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6zf7\" (UniqueName: \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.308772 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.308558    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.309215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.309194    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.310903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.310886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.329408 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.329354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6zf7\" (UniqueName: \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\") pod \"data-science-collector-collector-1\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.341060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.341026    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:48.472957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.472928    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:02:48.767782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.767752    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:02:48.771438 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:02:48.771407    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdaac8c_e288_4049_8566_04141fff6bfb.slice/crio-676d0c12f63a662e2736cda1521a760ce0e4fb1be42309f1683839a3cddebd92 WatchSource:0}: Error finding container 676d0c12f63a662e2736cda1521a760ce0e4fb1be42309f1683839a3cddebd92: Status 404 returned error can't find the container with id 676d0c12f63a662e2736cda1521a760ce0e4fb1be42309f1683839a3cddebd92
Oct 30 04:02:48.773527 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.773496    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:02:48.773613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:48.773550    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:02:49.633133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.633092    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ccdaac8c-e288-4049-8566-04141fff6bfb","Type":"ContainerStarted","Data":"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"}
Oct 30 04:02:49.633540 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.633162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ccdaac8c-e288-4049-8566-04141fff6bfb","Type":"ContainerStarted","Data":"676d0c12f63a662e2736cda1521a760ce0e4fb1be42309f1683839a3cddebd92"}
Oct 30 04:02:49.646213 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.646171    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:02:49.649738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.649711    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.662903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.662876    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 04:02:49.662903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.662892    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:02:49.663067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.662926    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 04:02:49.663067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.662932    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:02:49.665776 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.665758    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 04:02:49.665776 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.665770    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 04:02:49.665932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.665863    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-2cvcq\""
Oct 30 04:02:49.665932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.665902    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 04:02:49.678747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.678723    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:02:49.684810 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.684756    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.684355108 podStartE2EDuration="1.684355108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:49.683243772 +0000 UTC m=+4927.907493050" watchObservedRunningTime="2026-10-30 04:02:49.684355108 +0000 UTC m=+4927.908604389"
Oct 30 04:02:49.700641 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.700610    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:49.820567 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820533    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.820760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.820760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820592    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.820760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820640    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.820760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns6kb\" (UniqueName: \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.821014 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820891    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.821014 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820920    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.821014 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.820938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.884724 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.884638    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:02:49.921949 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.921916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.921958    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922211 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922187    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922251    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922307    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922324    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns6kb\" (UniqueName: \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922963    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.922991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.922978    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.923211 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.923059    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.923291 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.923269    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.924512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.924487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.924736 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.924719    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.924954 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.924933    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:49.944258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:49.944233    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns6kb\" (UniqueName: \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:55.974679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:55.974644    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.000086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.000051    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.075719 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.075680    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.075928 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.075802    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.088400 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.088374    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.187493 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.187459    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.212366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.212337    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.266227 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.266148    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:56.592107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.592028    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:02:56.595776 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:02:56.595745    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c12dc8_d9c1_421f_9567_e94a07122794.slice/crio-e05ebc52e0630fa142c9569a4e22fed66dcf7bde68205e0ba6e1b53f2d612561 WatchSource:0}: Error finding container e05ebc52e0630fa142c9569a4e22fed66dcf7bde68205e0ba6e1b53f2d612561: Status 404 returned error can't find the container with id e05ebc52e0630fa142c9569a4e22fed66dcf7bde68205e0ba6e1b53f2d612561
Oct 30 04:02:56.663222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.663188    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerStarted","Data":"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"}
Oct 30 04:02:56.663350 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:56.663260    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerStarted","Data":"e05ebc52e0630fa142c9569a4e22fed66dcf7bde68205e0ba6e1b53f2d612561"}
Oct 30 04:02:57.670130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.670093    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerStarted","Data":"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"}
Oct 30 04:02:57.670613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.670148    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerStarted","Data":"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"}
Oct 30 04:02:57.670613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.670222    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:57.670613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.670395    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:57.670613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.670437    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:57.672932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.672909    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:57.678799 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.678781    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:02:57.741407 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:02:57.740408    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=8.740392052 podStartE2EDuration="8.740392052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:57.727700482 +0000 UTC m=+4935.951949758" watchObservedRunningTime="2026-10-30 04:02:57.740392052 +0000 UTC m=+4935.964641329"
Oct 30 04:03:04.630414 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:04.630375    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:03:05.738112 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:05.738077    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:05.738546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:05.738313    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ccdaac8c-e288-4049-8566-04141fff6bfb" containerName="otc-container" containerID="cri-o://014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b" gracePeriod=30
Oct 30 04:03:05.807863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:05.807830    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:05.945172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:05.945139    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:05.989372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:05.989311    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:06.056297 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.056264    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\") pod \"ccdaac8c-e288-4049-8566-04141fff6bfb\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") "
Oct 30 04:03:06.056466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.056347    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\") pod \"ccdaac8c-e288-4049-8566-04141fff6bfb\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") "
Oct 30 04:03:06.056466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.056370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\" (UniqueName: \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\") pod \"ccdaac8c-e288-4049-8566-04141fff6bfb\" (UID: \"ccdaac8c-e288-4049-8566-04141fff6bfb\") "
Oct 30 04:03:06.059470 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.057050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal" pod "ccdaac8c-e288-4049-8566-04141fff6bfb" (UID: "ccdaac8c-e288-4049-8566-04141fff6bfb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:06.064853 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.059608    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7" pod "ccdaac8c-e288-4049-8566-04141fff6bfb" (UID: "ccdaac8c-e288-4049-8566-04141fff6bfb"). InnerVolumeSpecName "kube-api-access-c6zf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:06.064853 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.062335    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:06.065317 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.065276    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs" pod "ccdaac8c-e288-4049-8566-04141fff6bfb" (UID: "ccdaac8c-e288-4049-8566-04141fff6bfb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:06.158090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.158056    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6zf7\" (UniqueName: \"kubernetes.io/projected/ccdaac8c-e288-4049-8566-04141fff6bfb-kube-api-access-c6zf7\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:06.158090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.158084    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ccdaac8c-e288-4049-8566-04141fff6bfb-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:06.158090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.158094    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccdaac8c-e288-4049-8566-04141fff6bfb-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:06.216618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.216578    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:03:06.216954 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.216904    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo" containerID="cri-o://6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916" gracePeriod=30
Oct 30 04:03:06.216954 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.216920    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway-opa" containerID="cri-o://ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8" gracePeriod=30
Oct 30 04:03:06.217147 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.216928    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway" containerID="cri-o://dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa" gracePeriod=30
Oct 30 04:03:06.278676 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.278581    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:06.400770 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.400736    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:06.714209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.714176    4528 generic.go:356] "Generic (PLEG): container finished" podID="ccdaac8c-e288-4049-8566-04141fff6bfb" containerID="014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b" exitCode=0
Oct 30 04:03:06.714383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.714240    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:06.714383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.714259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ccdaac8c-e288-4049-8566-04141fff6bfb","Type":"ContainerDied","Data":"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"}
Oct 30 04:03:06.714383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.714315    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ccdaac8c-e288-4049-8566-04141fff6bfb","Type":"ContainerDied","Data":"676d0c12f63a662e2736cda1521a760ce0e4fb1be42309f1683839a3cddebd92"}
Oct 30 04:03:06.714383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.714344    4528 scope.go:122] "RemoveContainer" containerID="014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"
Oct 30 04:03:06.716510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.716488    4528 generic.go:356] "Generic (PLEG): container finished" podID="50c12dc8-d9c1-421f-9567-e94a07122794" containerID="ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8" exitCode=0
Oct 30 04:03:06.716510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.716508    4528 generic.go:356] "Generic (PLEG): container finished" podID="50c12dc8-d9c1-421f-9567-e94a07122794" containerID="6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916" exitCode=0
Oct 30 04:03:06.716670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.716543    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerDied","Data":"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"}
Oct 30 04:03:06.716670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.716583    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerDied","Data":"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"}
Oct 30 04:03:06.723235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.723221    4528 scope.go:122] "RemoveContainer" containerID="014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"
Oct 30 04:03:06.723490 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:06.723471    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b\": container with ID starting with 014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b not found: ID does not exist" containerID="014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"
Oct 30 04:03:06.723548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.723498    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b"} err="failed to get container status \"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b\": rpc error: code = NotFound desc = could not find container \"014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b\": container with ID starting with 014abaa8590ab44154d192ec11f190e56f8af29b969b755639fc4a0b00fcc09b not found: ID does not exist"
Oct 30 04:03:06.763331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.763308    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:06.766982 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:06.766960    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:07.454984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.454958    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:03:07.570779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570698    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.570779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570740    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.570779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570764    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570801    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\" (UniqueName: \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570849    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.570889    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.571013    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.571102    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.571122    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:07.571245 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.571190    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"50c12dc8-d9c1-421f-9567-e94a07122794\" (UID: \"50c12dc8-d9c1-421f-9567-e94a07122794\") "
Oct 30 04:03:07.571369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.571352    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.573207 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.572936    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:07.573410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.573278    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:07.573410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.573363    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:07.573559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.573536    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:07.573924 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.573901    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:03:07.574919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.574896    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:07.575216 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.575198    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "kube-api-access-ns6kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:07.575281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.575203    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants" pod "50c12dc8-d9c1-421f-9567-e94a07122794" (UID: "50c12dc8-d9c1-421f-9567-e94a07122794"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:07.671834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671793    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns6kb\" (UniqueName: \"kubernetes.io/projected/50c12dc8-d9c1-421f-9567-e94a07122794-kube-api-access-ns6kb\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671852    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671863    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671873    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671884    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671913    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671922    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.672009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.671931    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/50c12dc8-d9c1-421f-9567-e94a07122794-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:07.679955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.679934    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:03:07.723329 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.723294    4528 generic.go:356] "Generic (PLEG): container finished" podID="50c12dc8-d9c1-421f-9567-e94a07122794" containerID="dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa" exitCode=0
Oct 30 04:03:07.723501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.723349    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:03:07.723501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.723376    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerDied","Data":"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"}
Oct 30 04:03:07.723501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.723427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"50c12dc8-d9c1-421f-9567-e94a07122794","Type":"ContainerDied","Data":"e05ebc52e0630fa142c9569a4e22fed66dcf7bde68205e0ba6e1b53f2d612561"}
Oct 30 04:03:07.723501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.723455    4528 scope.go:122] "RemoveContainer" containerID="ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"
Oct 30 04:03:07.731503 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.731425    4528 scope.go:122] "RemoveContainer" containerID="dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"
Oct 30 04:03:07.738387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.738359    4528 scope.go:122] "RemoveContainer" containerID="6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"
Oct 30 04:03:07.745070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745055    4528 scope.go:122] "RemoveContainer" containerID="ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"
Oct 30 04:03:07.745320 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:07.745302    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8\": container with ID starting with ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8 not found: ID does not exist" containerID="ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"
Oct 30 04:03:07.745391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745326    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8"} err="failed to get container status \"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8\": rpc error: code = NotFound desc = could not find container \"ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8\": container with ID starting with ed230d7d4c4e0a0f6ad45d2296372f93211f400bafafa32e8cd103ce9c8a9da8 not found: ID does not exist"
Oct 30 04:03:07.745391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745343    4528 scope.go:122] "RemoveContainer" containerID="dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"
Oct 30 04:03:07.745605 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:07.745585    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa\": container with ID starting with dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa not found: ID does not exist" containerID="dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"
Oct 30 04:03:07.745646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745611    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa"} err="failed to get container status \"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa\": rpc error: code = NotFound desc = could not find container \"dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa\": container with ID starting with dd330f0ed0b0750060802950026abf722ab31f70d91afa8da2b1f2a330bfc9fa not found: ID does not exist"
Oct 30 04:03:07.745646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745625    4528 scope.go:122] "RemoveContainer" containerID="6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"
Oct 30 04:03:07.745859 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:07.745840    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916\": container with ID starting with 6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916 not found: ID does not exist" containerID="6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"
Oct 30 04:03:07.745927 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.745862    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916"} err="failed to get container status \"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916\": rpc error: code = NotFound desc = could not find container \"6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916\": container with ID starting with 6fc3fa8fa6ae513aa513034545cc58624de9e09694fc2e5afb5eb3d4c4f67916 not found: ID does not exist"
Oct 30 04:03:07.772345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.772322    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 04:03:07.773047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.773029    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:03:07.778888 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.778867    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:03:07.892313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.892281    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" path="/var/lib/kubelet/pods/50c12dc8-d9c1-421f-9567-e94a07122794/volumes"
Oct 30 04:03:07.892738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:07.892725    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccdaac8c-e288-4049-8566-04141fff6bfb" path="/var/lib/kubelet/pods/ccdaac8c-e288-4049-8566-04141fff6bfb/volumes"
Oct 30 04:03:09.826749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:09.826702    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:09.953643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:09.953612    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.075733 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.075698    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.189124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189085    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:03:10.189459 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189445    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway-opa"
Oct 30 04:03:10.189501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189462    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway-opa"
Oct 30 04:03:10.189501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189483    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway"
Oct 30 04:03:10.189501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189491    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway"
Oct 30 04:03:10.189595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189503    4528 cpu_manager.go:414] "containerMap: removing container" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo"
Oct 30 04:03:10.189595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189511    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo"
Oct 30 04:03:10.189595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189537    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ccdaac8c-e288-4049-8566-04141fff6bfb" containerName="otc-container"
Oct 30 04:03:10.189595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189544    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdaac8c-e288-4049-8566-04141fff6bfb" containerName="otc-container"
Oct 30 04:03:10.189712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189635    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway-opa"
Oct 30 04:03:10.189712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189647    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo-gateway"
Oct 30 04:03:10.189712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189656    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c12dc8-d9c1-421f-9567-e94a07122794" containerName="tempo"
Oct 30 04:03:10.189712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.189668    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdaac8c-e288-4049-8566-04141fff6bfb" containerName="otc-container"
Oct 30 04:03:10.194692 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.194670    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.205107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.205077    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.213476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.213452    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:03:10.228896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.228867    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:03:10.232507 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.232487    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.255192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.255161    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:03:10.267914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.267888    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:03:10.267914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.267906    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-c9m75\""
Oct 30 04:03:10.268102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.267926    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:03:10.268102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.267954    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:03:10.268102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.267975    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:03:10.268102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.268097    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:03:10.268288 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.268158    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:03:10.268288 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.268212    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:03:10.268387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.268334    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:03:10.268479 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.268451    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-hgwc4\""
Oct 30 04:03:10.293664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293635    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.293857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293667    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.293857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293694    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqw2d\" (UniqueName: \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.293857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293761    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.293857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293841    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.294067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293898    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.294067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.293930    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trbz\" (UniqueName: \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.294067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.294007    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.368596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.368554    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.394543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394514    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.394664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394552    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.394664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394628    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqw2d\" (UniqueName: \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.394664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394660    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.394950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394688    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.394950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394719    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.394950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394758    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trbz\" (UniqueName: \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.394950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.394783    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.395394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.395367    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.395596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.395578    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.395660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.395595    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.395660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.395624    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.397057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.397039    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.397373 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.397358    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.417267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.417240    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trbz\" (UniqueName: \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-hd7n8\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.417393 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.417326    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqw2d\" (UniqueName: \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-8ff6v\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.512395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.512313    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:03:10.547378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.547344    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:03:10.809233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.809207    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:03:10.811574 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:03:10.811549    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987143a2_8d43_4e69_bfd8_e3b3ca7b60e5.slice/crio-e5070a0a0e9d9242affd88fa3186934bc525f4be730c162350c87399a6086a62 WatchSource:0}: Error finding container e5070a0a0e9d9242affd88fa3186934bc525f4be730c162350c87399a6086a62: Status 404 returned error can't find the container with id e5070a0a0e9d9242affd88fa3186934bc525f4be730c162350c87399a6086a62
Oct 30 04:03:10.834332 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.834307    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:10.847564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:10.847539    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:03:10.849742 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:03:10.849721    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941da6d5_0d51_4605_bb24_788f7d04065a.slice/crio-db9e8285c8937e636a1b0aa19997848df40dae1def65a2687301210c15aa0a4c WatchSource:0}: Error finding container db9e8285c8937e636a1b0aa19997848df40dae1def65a2687301210c15aa0a4c: Status 404 returned error can't find the container with id db9e8285c8937e636a1b0aa19997848df40dae1def65a2687301210c15aa0a4c
Oct 30 04:03:11.011968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.011933    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:11.202240 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.202206    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:11.745665 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.745626    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerStarted","Data":"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"}
Oct 30 04:03:11.745871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.745703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerStarted","Data":"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"}
Oct 30 04:03:11.745871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.745740    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerStarted","Data":"db9e8285c8937e636a1b0aa19997848df40dae1def65a2687301210c15aa0a4c"}
Oct 30 04:03:11.747974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.747944    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerStarted","Data":"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"}
Oct 30 04:03:11.748090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.748004    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerStarted","Data":"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"}
Oct 30 04:03:11.748090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.748041    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerStarted","Data":"e5070a0a0e9d9242affd88fa3186934bc525f4be730c162350c87399a6086a62"}
Oct 30 04:03:11.786275 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.786216    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" podStartSLOduration=1.786199221 podStartE2EDuration="1.786199221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:11.785102033 +0000 UTC m=+4950.009351333" watchObservedRunningTime="2026-10-30 04:03:11.786199221 +0000 UTC m=+4950.010448503"
Oct 30 04:03:11.810118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.810083    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:11.814156 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.814133    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.827893 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.827802    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" podStartSLOduration=1.827787466 podStartE2EDuration="1.827787466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:11.827223912 +0000 UTC m=+4950.051473190" watchObservedRunningTime="2026-10-30 04:03:11.827787466 +0000 UTC m=+4950.052036740"
Oct 30 04:03:11.829570 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.829482    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:11.837410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.837377    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:11.847696 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.847672    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:03:11.882792 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.882759    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\""
Oct 30 04:03:11.882958 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.882794    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:03:11.882958 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.882803    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:03:11.882958 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.882794    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:03:11.907230 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.907200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.907396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.907240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.907396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.907329    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.907396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.907382    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jdv\" (UniqueName: \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.907605 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.907485    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:11.947058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:11.947026    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:12.008276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.008197    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.008276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.008246    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.008276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.008275    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.008548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.008298    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.008548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.008316    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jdv\" (UniqueName: \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.009061 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.009037    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.010878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.010849    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.010975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.010856    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.010975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.010958    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.030985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.030961    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jdv\" (UniqueName: \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\") pod \"data-science-collector-collector-1\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.062873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.062833    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:12.129225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.129195    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:12.427667 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.427489    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:12.430232 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:03:12.430205    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c7202f_13f6_4365_b3bc_aec4f749a411.slice/crio-61ff4f3ca82da49c43666ded43a06f79f745fef68c8da91a197909c2d5af6a12 WatchSource:0}: Error finding container 61ff4f3ca82da49c43666ded43a06f79f745fef68c8da91a197909c2d5af6a12: Status 404 returned error can't find the container with id 61ff4f3ca82da49c43666ded43a06f79f745fef68c8da91a197909c2d5af6a12
Oct 30 04:03:12.432489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.432459    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:03:12.432557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.432515    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:03:12.754314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.754222    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"43c7202f-13f6-4365-b3bc-aec4f749a411","Type":"ContainerStarted","Data":"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"}
Oct 30 04:03:12.754314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.754283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"43c7202f-13f6-4365-b3bc-aec4f749a411","Type":"ContainerStarted","Data":"61ff4f3ca82da49c43666ded43a06f79f745fef68c8da91a197909c2d5af6a12"}
Oct 30 04:03:12.795553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:12.795500    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.795485811 podStartE2EDuration="1.795485811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:12.794993152 +0000 UTC m=+4951.019242441" watchObservedRunningTime="2026-10-30 04:03:12.795485811 +0000 UTC m=+4951.019735149"
Oct 30 04:03:14.775296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:14.775259    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.017729 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.017687    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.128192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.128104    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.250765 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.250732    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.409244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.409208    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.552571 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.552538    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:03:15.556469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.556446    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.571432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.571396    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:15.572800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.572777    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:03:15.573279 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.573247    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:03:15.575698 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.575678    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:03:15.617090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617060    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:03:15.617340 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617322    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:03:15.617430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617398    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:03:15.617430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:03:15.617430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617417    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:03:15.617535 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617475    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5fhss\""
Oct 30 04:03:15.617958 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.617943    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:03:15.640406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640374    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640533 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640410    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640533 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640432    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvts\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640569    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640598    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640627    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640722    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640781 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640837 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640789    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640888 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.640925 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.640896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.673998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.673938    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:03:15.741911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.741878    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.741916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvts\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.741955    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.741975    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.741999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742028    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742086    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742317    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742356    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742532 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742412    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.742532 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742461    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.743102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.742747    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.743551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.743523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745260    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745309    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745505    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745555    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.745896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.746008 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.745989    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.746094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.746077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:15.764589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:15.764566    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvts\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:17.961644 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:17.961611    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:17.989466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:17.989430    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.061974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.061941    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.062171 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.062092    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.075683 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.075655    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.175003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.174973    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.200596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.200568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.279626 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.279540    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:18.615164 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:03:18.615132    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b105965_1079_4f5e_9f5a_b4f2952996e7.slice/crio-cd5477a2cf4d1786fc7ebbe61ed6d9cf6833c7ad17e070645f032b287cfe0559 WatchSource:0}: Error finding container cd5477a2cf4d1786fc7ebbe61ed6d9cf6833c7ad17e070645f032b287cfe0559: Status 404 returned error can't find the container with id cd5477a2cf4d1786fc7ebbe61ed6d9cf6833c7ad17e070645f032b287cfe0559
Oct 30 04:03:18.624421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.624397    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:03:18.781294 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:18.781256    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerStarted","Data":"cd5477a2cf4d1786fc7ebbe61ed6d9cf6833c7ad17e070645f032b287cfe0559"}
Oct 30 04:03:19.786770 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:19.786734    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerStarted","Data":"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"}
Oct 30 04:03:22.801446 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:22.801355    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerID="fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c" exitCode=0
Oct 30 04:03:22.801446 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:22.801427    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerDied","Data":"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"}
Oct 30 04:03:23.807902 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:23.807869    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerStarted","Data":"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"}
Oct 30 04:03:23.808284 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:23.807939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerStarted","Data":"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"}
Oct 30 04:03:23.808284 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:23.807965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerStarted","Data":"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"}
Oct 30 04:03:23.882667 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:23.882612    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=5.39606238 podStartE2EDuration="8.882596808s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.486534428s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:23.881765924 +0000 UTC m=+4962.106015200" watchObservedRunningTime="2026-10-30 04:03:23.882596808 +0000 UTC m=+4962.106846085"
Oct 30 04:03:28.279955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:28.279912    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:33.279681 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:33.279643    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:33.285042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:33.285015    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:33.853650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:33.853622    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:03:34.631022 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:34.630983    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:03:39.858159 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:39.858122    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:40.078721 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:40.078668    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:40.200025 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:40.199987    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:40.319232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:40.319201    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:40.520840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:40.520743    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:41.954705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:41.954672    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:41.955104 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:41.954873    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="43c7202f-13f6-4365-b3bc-aec4f749a411" containerName="otc-container" containerID="cri-o://c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1" gracePeriod=30
Oct 30 04:03:42.106121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:42.106021    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:42.220982 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:42.220943    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:42.347495 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:42.347464    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:44.114832 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.114786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:44.288589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.288561    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:44.361005 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.360936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\") pod \"43c7202f-13f6-4365-b3bc-aec4f749a411\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") "
Oct 30 04:03:44.361005 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.360976    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\") pod \"43c7202f-13f6-4365-b3bc-aec4f749a411\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") "
Oct 30 04:03:44.361226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.361016    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\") pod \"43c7202f-13f6-4365-b3bc-aec4f749a411\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") "
Oct 30 04:03:44.361226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.361043    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\" (UniqueName: \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\") pod \"43c7202f-13f6-4365-b3bc-aec4f749a411\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") "
Oct 30 04:03:44.361460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.361425    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal" pod "43c7202f-13f6-4365-b3bc-aec4f749a411" (UID: "43c7202f-13f6-4365-b3bc-aec4f749a411"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:44.363173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.363145    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert" pod "43c7202f-13f6-4365-b3bc-aec4f749a411" (UID: "43c7202f-13f6-4365-b3bc-aec4f749a411"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:44.363173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.363167    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs" pod "43c7202f-13f6-4365-b3bc-aec4f749a411" (UID: "43c7202f-13f6-4365-b3bc-aec4f749a411"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:44.363314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.363194    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv" pod "43c7202f-13f6-4365-b3bc-aec4f749a411" (UID: "43c7202f-13f6-4365-b3bc-aec4f749a411"). InnerVolumeSpecName "kube-api-access-99jdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:44.462074 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.462041    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\") pod \"43c7202f-13f6-4365-b3bc-aec4f749a411\" (UID: \"43c7202f-13f6-4365-b3bc-aec4f749a411\") "
Oct 30 04:03:44.462244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.462216    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:44.462244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.462226    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c7202f-13f6-4365-b3bc-aec4f749a411-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:44.462244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.462236    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:44.462429 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.462247    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jdv\" (UniqueName: \"kubernetes.io/projected/43c7202f-13f6-4365-b3bc-aec4f749a411-kube-api-access-99jdv\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:44.464105 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.464080    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs" pod "43c7202f-13f6-4365-b3bc-aec4f749a411" (UID: "43c7202f-13f6-4365-b3bc-aec4f749a411"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:44.563169 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.563123    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c7202f-13f6-4365-b3bc-aec4f749a411-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:03:44.897189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.897155    4528 generic.go:356] "Generic (PLEG): container finished" podID="43c7202f-13f6-4365-b3bc-aec4f749a411" containerID="c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1" exitCode=0
Oct 30 04:03:44.897371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.897206    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:44.897371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.897227    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"43c7202f-13f6-4365-b3bc-aec4f749a411","Type":"ContainerDied","Data":"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"}
Oct 30 04:03:44.897371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.897265    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"43c7202f-13f6-4365-b3bc-aec4f749a411","Type":"ContainerDied","Data":"61ff4f3ca82da49c43666ded43a06f79f745fef68c8da91a197909c2d5af6a12"}
Oct 30 04:03:44.897371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.897285    4528 scope.go:122] "RemoveContainer" containerID="c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"
Oct 30 04:03:44.909427 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.909408    4528 scope.go:122] "RemoveContainer" containerID="c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"
Oct 30 04:03:44.909895 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:03:44.909874    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1\": container with ID starting with c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1 not found: ID does not exist" containerID="c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"
Oct 30 04:03:44.909961 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.909905    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1"} err="failed to get container status \"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1\": rpc error: code = NotFound desc = could not find container \"c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1\": container with ID starting with c2f750e9ce891fdf138f8133957bf8dd2035a94f5ea1d7f2b7022da757a914d1 not found: ID does not exist"
Oct 30 04:03:44.947074 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.947038    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:44.958603 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.958573    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:44.988356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.988323    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:44.988673 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.988660    4528 cpu_manager.go:414] "containerMap: removing container" podUID="43c7202f-13f6-4365-b3bc-aec4f749a411" containerName="otc-container"
Oct 30 04:03:44.988733 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.988674    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c7202f-13f6-4365-b3bc-aec4f749a411" containerName="otc-container"
Oct 30 04:03:44.988808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.988741    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c7202f-13f6-4365-b3bc-aec4f749a411" containerName="otc-container"
Oct 30 04:03:44.993072 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:44.993053    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.009562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.009534    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-45ch8\""
Oct 30 04:03:45.009700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.009601    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:03:45.009700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.009612    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:03:45.009700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.009602    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:03:45.009700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.009660    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:03:45.022712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.022687    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:45.066492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.066463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.066648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.066498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.066648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.066581    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8s4\" (UniqueName: \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.066648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.066634    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.066754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.066678    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.167801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.167710    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.167801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.167779    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8s4\" (UniqueName: \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.167801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.167802    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.168338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.167860    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.168338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.167895    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.168548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.168528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.170294 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.170266    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.170509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.170484    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.170551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.170490    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.190448 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.190424    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8s4\" (UniqueName: \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\") pod \"data-science-collector-collector-1\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.305253 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.305215    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:03:45.603965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.603890    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:03:45.606982 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:03:45.606956    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1718522c_7007_4a2b_8539_a9c28c699d9a.slice/crio-a74cdc51c7fbf3dd4e2c7e55aebc3a187ea097ebcf6201437e417a89b17a8ee9 WatchSource:0}: Error finding container a74cdc51c7fbf3dd4e2c7e55aebc3a187ea097ebcf6201437e417a89b17a8ee9: Status 404 returned error can't find the container with id a74cdc51c7fbf3dd4e2c7e55aebc3a187ea097ebcf6201437e417a89b17a8ee9
Oct 30 04:03:45.609110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.609075    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:03:45.609196 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.609137    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:03:45.891750 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.891709    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c7202f-13f6-4365-b3bc-aec4f749a411" path="/var/lib/kubelet/pods/43c7202f-13f6-4365-b3bc-aec4f749a411/volumes"
Oct 30 04:03:45.903380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.903348    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1718522c-7007-4a2b-8539-a9c28c699d9a","Type":"ContainerStarted","Data":"7336c20c4d2d1377d7cf0788a7b531d76bfe613c4e5a5e69bbb3c8759ae933c5"}
Oct 30 04:03:45.903531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.903410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1718522c-7007-4a2b-8539-a9c28c699d9a","Type":"ContainerStarted","Data":"a74cdc51c7fbf3dd4e2c7e55aebc3a187ea097ebcf6201437e417a89b17a8ee9"}
Oct 30 04:03:45.945614 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:45.945548    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.945529114 podStartE2EDuration="1.945529114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:45.943566664 +0000 UTC m=+4984.167815942" watchObservedRunningTime="2026-10-30 04:03:45.945529114 +0000 UTC m=+4984.169778391"
Oct 30 04:03:52.957909 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:52.957862    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:03:53.076974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:03:53.076941    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:01.133082 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:01.132998    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:04.631443 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:04.631407    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:04:05.925186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:05.925148    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:06.043396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:06.043355    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:06.164897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:06.164859    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:06.280266 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:06.280186    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:07.384462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:07.384421    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:07.504100 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:07.504061    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:07.868834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:07.868734    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:04:07.869001 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:07.868980    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1718522c-7007-4a2b-8539-a9c28c699d9a" containerName="otc-container" containerID="cri-o://7336c20c4d2d1377d7cf0788a7b531d76bfe613c4e5a5e69bbb3c8759ae933c5" gracePeriod=30
Oct 30 04:04:08.009385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:08.009351    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:08.124014 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:08.123938    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:08.246028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:08.245988    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:09.210926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.210890    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:09.214476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.214454    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.228676 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.228651    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 04:04:09.228676 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.228666    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 04:04:09.228887 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.228664    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:04:09.228887 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.228651    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 04:04:09.231484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.231465    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:04:09.240209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.240192    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 04:04:09.243393 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.243371    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:09.260027 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.260003    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 04:04:09.268568 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.268550    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-ck8d6\""
Oct 30 04:04:09.276070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.276048    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:09.278621 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278597    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278632    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278660    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278790 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278743    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhnd\" (UniqueName: \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278790 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278872 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278872 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278832    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.278940 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.278887    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.379946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.379910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.379957    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.379972    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.379989    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.380044    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.380070    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380408 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.380231    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.380408 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.380290    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhnd\" (UniqueName: \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.381058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.381032    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.381184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.381106    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.381229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.381181    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.381293 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.381223    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.382635 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.382613    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.382635 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.382629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.382758 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.382685    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:09.399289 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.399266    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:09.402737 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:09.402719    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhnd\" (UniqueName: \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:10.004105 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.004074    4528 generic.go:356] "Generic (PLEG): container finished" podID="1718522c-7007-4a2b-8539-a9c28c699d9a" containerID="7336c20c4d2d1377d7cf0788a7b531d76bfe613c4e5a5e69bbb3c8759ae933c5" exitCode=0
Oct 30 04:04:10.004268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.004123    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1718522c-7007-4a2b-8539-a9c28c699d9a","Type":"ContainerDied","Data":"7336c20c4d2d1377d7cf0788a7b531d76bfe613c4e5a5e69bbb3c8759ae933c5"}
Oct 30 04:04:10.114215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.114190    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:04:10.187778 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.187740    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\") pod \"1718522c-7007-4a2b-8539-a9c28c699d9a\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") "
Oct 30 04:04:10.187984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.187795    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\") pod \"1718522c-7007-4a2b-8539-a9c28c699d9a\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") "
Oct 30 04:04:10.187984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.187849    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\") pod \"1718522c-7007-4a2b-8539-a9c28c699d9a\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") "
Oct 30 04:04:10.187984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.187881    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\" (UniqueName: \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\") pod \"1718522c-7007-4a2b-8539-a9c28c699d9a\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") "
Oct 30 04:04:10.187984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.187958    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\") pod \"1718522c-7007-4a2b-8539-a9c28c699d9a\" (UID: \"1718522c-7007-4a2b-8539-a9c28c699d9a\") "
Oct 30 04:04:10.188403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.188375    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal" pod "1718522c-7007-4a2b-8539-a9c28c699d9a" (UID: "1718522c-7007-4a2b-8539-a9c28c699d9a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:10.190064 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.190032    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs" pod "1718522c-7007-4a2b-8539-a9c28c699d9a" (UID: "1718522c-7007-4a2b-8539-a9c28c699d9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:10.190180 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.190096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert" pod "1718522c-7007-4a2b-8539-a9c28c699d9a" (UID: "1718522c-7007-4a2b-8539-a9c28c699d9a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:10.190326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.190307    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs" pod "1718522c-7007-4a2b-8539-a9c28c699d9a" (UID: "1718522c-7007-4a2b-8539-a9c28c699d9a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:10.190431 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.190413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4" pod "1718522c-7007-4a2b-8539-a9c28c699d9a" (UID: "1718522c-7007-4a2b-8539-a9c28c699d9a"). InnerVolumeSpecName "kube-api-access-lv8s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:10.289268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.289191    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1718522c-7007-4a2b-8539-a9c28c699d9a-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:10.289268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.289218    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:10.289268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.289229    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:10.289268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.289239    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1718522c-7007-4a2b-8539-a9c28c699d9a-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:10.289268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:10.289248    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8s4\" (UniqueName: \"kubernetes.io/projected/1718522c-7007-4a2b-8539-a9c28c699d9a-kube-api-access-lv8s4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:11.009467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.009389    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:04:11.009620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.009386    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1718522c-7007-4a2b-8539-a9c28c699d9a","Type":"ContainerDied","Data":"a74cdc51c7fbf3dd4e2c7e55aebc3a187ea097ebcf6201437e417a89b17a8ee9"}
Oct 30 04:04:11.009620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.009525    4528 scope.go:122] "RemoveContainer" containerID="7336c20c4d2d1377d7cf0788a7b531d76bfe613c4e5a5e69bbb3c8759ae933c5"
Oct 30 04:04:11.059594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.059559    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:04:11.070981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.070956    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:04:11.891746 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:11.891713    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1718522c-7007-4a2b-8539-a9c28c699d9a" path="/var/lib/kubelet/pods/1718522c-7007-4a2b-8539-a9c28c699d9a/volumes"
Oct 30 04:04:14.793097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:14.793058    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:14.907807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:14.907770    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:15.106536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:15.106449    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:15.228142 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:15.228108    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:16.806468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:16.806435    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:16.922902 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:16.922842    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:17.057216 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.057131    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:17.754447 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.754408    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.783969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.783934    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.803020 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.802984    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:17.803316 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:17.803294    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[tempo-storage], unattached volumes=[], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="3f1ee506-c480-47d2-8af2-49695dff95ef"
Oct 30 04:04:17.855077 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.855039    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.855466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.855189    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.866882 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.866851    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:17.868930 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.868909    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.972120 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.972090    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:17.998934 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:17.998905    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:18.038672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.038591    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:18.046797 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.046774    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:18.158737 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158702    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.158919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158844    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.158919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\" (UniqueName: \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.158919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158909    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.159081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158935    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.159081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.158977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.159682 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.159646    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.159888 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.159859    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:18.160088 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.159972    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:18.160233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.159872    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.160378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.160363    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\") pod \"3f1ee506-c480-47d2-8af2-49695dff95ef\" (UID: \"3f1ee506-c480-47d2-8af2-49695dff95ef\") "
Oct 30 04:04:18.161834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.160330    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:18.161834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.161499    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:18.162500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.162472    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.162617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.162502    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.162617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.162518    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.162617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.162542    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.169598 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.169548    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:18.169728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.169611    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "kube-api-access-pkhnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:18.169728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.169640    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:18.169879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.169840    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:04:18.169999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.169984    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants" pod "3f1ee506-c480-47d2-8af2-49695dff95ef" (UID: "3f1ee506-c480-47d2-8af2-49695dff95ef"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:18.263659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.263621    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.263864 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.263671    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:04:18.263864 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.263685    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkhnd\" (UniqueName: \"kubernetes.io/projected/3f1ee506-c480-47d2-8af2-49695dff95ef-kube-api-access-pkhnd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.263864 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.263695    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.263864 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.263707    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/3f1ee506-c480-47d2-8af2-49695dff95ef-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:18.275308 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.275285    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:04:18.365006 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:18.364929    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 04:04:19.042596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:19.042567    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:19.120810 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:19.120770    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:19.124581 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:19.124554    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:19.892494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:19.892459    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1ee506-c480-47d2-8af2-49695dff95ef" path="/var/lib/kubelet/pods/3f1ee506-c480-47d2-8af2-49695dff95ef/volumes"
Oct 30 04:04:23.862189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:23.862159    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:23.983898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:23.983861    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:24.095025 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:24.094986    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:24.208974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:24.208947    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:25.963504 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:25.963458    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:26.092700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.092670    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:26.224383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.224302    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:26.811740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.811708    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:26.812073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.812059    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1718522c-7007-4a2b-8539-a9c28c699d9a" containerName="otc-container"
Oct 30 04:04:26.812120 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.812076    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1718522c-7007-4a2b-8539-a9c28c699d9a" containerName="otc-container"
Oct 30 04:04:26.812187 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.812174    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1718522c-7007-4a2b-8539-a9c28c699d9a" containerName="otc-container"
Oct 30 04:04:26.815598 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.815575    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.831372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.831350    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:04:26.842791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842748    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\""
Oct 30 04:04:26.842936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842798    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\""
Oct 30 04:04:26.842936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842881    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\""
Oct 30 04:04:26.842936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842897    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\""
Oct 30 04:04:26.843059 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842950    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\""
Oct 30 04:04:26.843059 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.842967    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\""
Oct 30 04:04:26.843059 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.843051    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-85hzf\""
Oct 30 04:04:26.844280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.844254    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:26.875384 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.875352    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:26.938166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938132    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938180    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938253    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938308    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wtk2\" (UniqueName: \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938352    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938379    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938426    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:26.938498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:26.938465    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.007590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.007551    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:27.039206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039179    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039218    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wtk2\" (UniqueName: \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039310    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039339    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039358    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039384    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.039587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.039434    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.040068 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.040036    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.040237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.040213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.040293 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.040252    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.040413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.040388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.042186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.042163    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.042308 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.042244    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.042611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.042587    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:27.062300 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:27.062249    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wtk2\" (UniqueName: \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.048969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.048938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.088884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.088846    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.149332 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.149298    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.149499 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.149408    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.163210 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.163180    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.268200 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.268171    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5f08952af3c1047a3ba241d606ea79a8188c0f50e87620e0339bd35bd6285b1/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.294474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.294444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.331712 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.331639    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:28.660359 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:28.660335    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:28.662951 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:04:28.662923    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3741d9_c061_4a50_9038_7f9bc63634d5.slice/crio-d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359 WatchSource:0}: Error finding container d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359: Status 404 returned error can't find the container with id d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359
Oct 30 04:04:29.088224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088136    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerStarted","Data":"cd6ec47cc11538cf5376518fd060ac10c3f2583218eee29f143d3d282fc8e53c"}
Oct 30 04:04:29.088224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088194    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerStarted","Data":"a7dd66f91f71656cfca6f97735889ad6c46b86a70a05fec33102c89b228b51ba"}
Oct 30 04:04:29.088750 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088224    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:29.088750 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088354    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:29.088750 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerStarted","Data":"38989f577be8d4a7c3a7136042d14cdefdc4773277195c4936c133885dabde4b"}
Oct 30 04:04:29.088750 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.088435    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerStarted","Data":"d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359"}
Oct 30 04:04:29.091077 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.091056    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:29.096769 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.096751    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:29.134432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:29.134385    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=3.134372695 podStartE2EDuration="3.134372695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:29.132594682 +0000 UTC m=+5027.356843959" watchObservedRunningTime="2026-10-30 04:04:29.134372695 +0000 UTC m=+5027.358621972"
Oct 30 04:04:30.093024 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:30.092991    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:32.989305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:32.989261    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:33.185916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:33.185881    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:33.297420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:33.297340    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:34.631233 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:34.631200    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:04:34.959340 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:34.959240    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:35.070687 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.070653    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:35.182760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.182729    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:35.885027 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.884994    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:35.885454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.885247    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo" containerID="cri-o://38989f577be8d4a7c3a7136042d14cdefdc4773277195c4936c133885dabde4b" gracePeriod=30
Oct 30 04:04:35.885454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.885308    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway" containerID="cri-o://a7dd66f91f71656cfca6f97735889ad6c46b86a70a05fec33102c89b228b51ba" gracePeriod=30
Oct 30 04:04:35.885454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.885308    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway-opa" containerID="cri-o://cd6ec47cc11538cf5376518fd060ac10c3f2583218eee29f143d3d282fc8e53c" gracePeriod=30
Oct 30 04:04:35.946563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:35.946511    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:36.118465 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:36.118436    4528 generic.go:356] "Generic (PLEG): container finished" podID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerID="cd6ec47cc11538cf5376518fd060ac10c3f2583218eee29f143d3d282fc8e53c" exitCode=0
Oct 30 04:04:36.118465 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:36.118459    4528 generic.go:356] "Generic (PLEG): container finished" podID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerID="38989f577be8d4a7c3a7136042d14cdefdc4773277195c4936c133885dabde4b" exitCode=0
Oct 30 04:04:36.118672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:36.118508    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerDied","Data":"cd6ec47cc11538cf5376518fd060ac10c3f2583218eee29f143d3d282fc8e53c"}
Oct 30 04:04:36.118672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:36.118567    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerDied","Data":"38989f577be8d4a7c3a7136042d14cdefdc4773277195c4936c133885dabde4b"}
Oct 30 04:04:37.125172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.125143    4528 generic.go:356] "Generic (PLEG): container finished" podID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerID="a7dd66f91f71656cfca6f97735889ad6c46b86a70a05fec33102c89b228b51ba" exitCode=0
Oct 30 04:04:37.125481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.125168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerDied","Data":"a7dd66f91f71656cfca6f97735889ad6c46b86a70a05fec33102c89b228b51ba"}
Oct 30 04:04:37.125481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.125234    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"cf3741d9-c061-4a50-9038-7f9bc63634d5","Type":"ContainerDied","Data":"d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359"}
Oct 30 04:04:37.125481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.125262    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1424dadd01342622866eb6471cd42a61ada46f578cbfc1770049bedb6149359"
Oct 30 04:04:37.138729 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.138707    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:37.223863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.223751    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224039 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.223891    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224039 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.223947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224039 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.223965    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224039 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.223992    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224042    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224077    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\" (UniqueName: \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224148    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\") pod \"cf3741d9-c061-4a50-9038-7f9bc63634d5\" (UID: \"cf3741d9-c061-4a50-9038-7f9bc63634d5\") "
Oct 30 04:04:37.224479 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224317    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:37.224479 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224362    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:37.224595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224483    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-conf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.224595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224507    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.224857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224800    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:37.224857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.224841    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:37.226366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.226343    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:37.226492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.226421    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:37.226562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.226551    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:37.226895 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.226857    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "kube-api-access-4wtk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:37.227224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.227206    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9" pod "cf3741d9-c061-4a50-9038-7f9bc63634d5" (UID: "cf3741d9-c061-4a50-9038-7f9bc63634d5"). InnerVolumeSpecName "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:04:37.325114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325077    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325107    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325117    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wtk2\" (UniqueName: \"kubernetes.io/projected/cf3741d9-c061-4a50-9038-7f9bc63634d5-kube-api-access-4wtk2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325368 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325126    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325368 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325136    4528 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-gateway-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325368 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325146    4528 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/cf3741d9-c061-4a50-9038-7f9bc63634d5-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:37.325368 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.325176    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:04:37.334546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.334527    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:04:37.426229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:37.426188    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-4f4b8000-90a8-496d-bbe6-5ab177248e69\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09c15f678a0743eb9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-be9ce8daef5954efc86e9393f82ad1805283dca9a35857661fb5d63ffe640743\""
Oct 30 04:04:38.129730 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:38.129692    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"
Oct 30 04:04:38.191879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:38.191776    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:38.203154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:38.203127    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"]
Oct 30 04:04:39.893811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:39.893775    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" path="/var/lib/kubelet/pods/cf3741d9-c061-4a50-9038-7f9bc63634d5/volumes"
Oct 30 04:04:39.987715 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:39.987676    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:40.114414 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.114383    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:04:40.114610 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.114590    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="kube-rbac-proxy" containerID="cri-o://20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c" gracePeriod=30
Oct 30 04:04:40.114683 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.114664    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="prom-label-proxy" containerID="cri-o://0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8" gracePeriod=30
Oct 30 04:04:40.135045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.135014    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:04:40.135329 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.135301    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="kube-rbac-proxy" containerID="cri-o://8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5" gracePeriod=30
Oct 30 04:04:40.135420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.135321    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="prom-label-proxy" containerID="cri-o://2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c" gracePeriod=30
Oct 30 04:04:40.392705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.392677    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:04:40.400503 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.400475    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:04:40.453654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453564    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\" (UniqueName: \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\") pod \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") "
Oct 30 04:04:40.453654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453619    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\") pod \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453667    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\" (UniqueName: \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\") pod \"941da6d5-0d51-4605-bb24-788f7d04065a\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453720    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\" (UniqueName: \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\") pod \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453794    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\") pod \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453835    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\") pod \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\" (UID: \"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\" (UniqueName: \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\") pod \"941da6d5-0d51-4605-bb24-788f7d04065a\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") "
Oct 30 04:04:40.453955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.453918    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\" (UniqueName: \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\") pod \"941da6d5-0d51-4605-bb24-788f7d04065a\" (UID: \"941da6d5-0d51-4605-bb24-788f7d04065a\") "
Oct 30 04:04:40.454242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.454107    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca" pod "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" (UID: "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:40.454242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.454192    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-prometheus-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.454641 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.454430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config" pod "941da6d5-0d51-4605-bb24-788f7d04065a" (UID: "941da6d5-0d51-4605-bb24-788f7d04065a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:40.455200 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.455160    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca" pod "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" (UID: "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:40.455417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.455397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config" pod "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" (UID: "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:40.458245 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.458220    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls" pod "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" (UID: "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:40.458709 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.458306    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz" pod "941da6d5-0d51-4605-bb24-788f7d04065a" (UID: "941da6d5-0d51-4605-bb24-788f7d04065a"). InnerVolumeSpecName "kube-api-access-9trbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:40.459236 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.459214    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls" pod "941da6d5-0d51-4605-bb24-788f7d04065a" (UID: "941da6d5-0d51-4605-bb24-788f7d04065a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:40.459656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.459581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d" pod "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" (UID: "987143a2-8d43-4e69-bfd8-e3b3ca7b60e5"). InnerVolumeSpecName "kube-api-access-kqw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:40.564844 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564769    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trbz\" (UniqueName: \"kubernetes.io/projected/941da6d5-0d51-4605-bb24-788f7d04065a-kube-api-access-9trbz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.564844 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564803    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/941da6d5-0d51-4605-bb24-788f7d04065a-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.564844 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564846    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqw2d\" (UniqueName: \"kubernetes.io/projected/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-kube-api-access-kqw2d\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.565126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564859    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/941da6d5-0d51-4605-bb24-788f7d04065a-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.565126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564876    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.565126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564886    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.565126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.564897    4528 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:40.589574 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.589527    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:40.678348 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.678317    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:04:40.678590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.678549    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="prometheus" containerID="cri-o://355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5" gracePeriod=600
Oct 30 04:04:40.678686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.678597    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="thanos-sidecar" containerID="cri-o://4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74" gracePeriod=600
Oct 30 04:04:40.678686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.678597    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="config-reloader" containerID="cri-o://965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548" gracePeriod=600
Oct 30 04:04:40.721214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.721138    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:40.829936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:40.829902    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:41.144645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144620    4528 generic.go:356] "Generic (PLEG): container finished" podID="941da6d5-0d51-4605-bb24-788f7d04065a" containerID="2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c" exitCode=0
Oct 30 04:04:41.144645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144640    4528 generic.go:356] "Generic (PLEG): container finished" podID="941da6d5-0d51-4605-bb24-788f7d04065a" containerID="8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5" exitCode=0
Oct 30 04:04:41.145079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144671    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"
Oct 30 04:04:41.145079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144700    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerDied","Data":"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"}
Oct 30 04:04:41.145079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144755    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerDied","Data":"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"}
Oct 30 04:04:41.145079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144777    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8" event={"ID":"941da6d5-0d51-4605-bb24-788f7d04065a","Type":"ContainerDied","Data":"db9e8285c8937e636a1b0aa19997848df40dae1def65a2687301210c15aa0a4c"}
Oct 30 04:04:41.145079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.144811    4528 scope.go:122] "RemoveContainer" containerID="2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"
Oct 30 04:04:41.146391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146371    4528 generic.go:356] "Generic (PLEG): container finished" podID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerID="0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8" exitCode=0
Oct 30 04:04:41.146391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146389    4528 generic.go:356] "Generic (PLEG): container finished" podID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerID="20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c" exitCode=0
Oct 30 04:04:41.146558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146427    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"
Oct 30 04:04:41.146558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146452    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerDied","Data":"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"}
Oct 30 04:04:41.146558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146515    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerDied","Data":"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"}
Oct 30 04:04:41.146558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.146548    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v" event={"ID":"987143a2-8d43-4e69-bfd8-e3b3ca7b60e5","Type":"ContainerDied","Data":"e5070a0a0e9d9242affd88fa3186934bc525f4be730c162350c87399a6086a62"}
Oct 30 04:04:41.149476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.149455    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerID="4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74" exitCode=0
Oct 30 04:04:41.149476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.149471    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerID="965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548" exitCode=0
Oct 30 04:04:41.149649 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.149487    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerDied","Data":"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"}
Oct 30 04:04:41.149649 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.149530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerDied","Data":"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"}
Oct 30 04:04:41.154404 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.154382    4528 scope.go:122] "RemoveContainer" containerID="8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"
Oct 30 04:04:41.161870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.161848    4528 scope.go:122] "RemoveContainer" containerID="2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"
Oct 30 04:04:41.162135 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:41.162116    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c\": container with ID starting with 2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c not found: ID does not exist" containerID="2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"
Oct 30 04:04:41.162195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162142    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"} err="failed to get container status \"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c\": rpc error: code = NotFound desc = could not find container \"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c\": container with ID starting with 2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c not found: ID does not exist"
Oct 30 04:04:41.162195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162160    4528 scope.go:122] "RemoveContainer" containerID="8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"
Oct 30 04:04:41.162385 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:41.162368    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5\": container with ID starting with 8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5 not found: ID does not exist" containerID="8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"
Oct 30 04:04:41.162444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162396    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"} err="failed to get container status \"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5\": rpc error: code = NotFound desc = could not find container \"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5\": container with ID starting with 8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5 not found: ID does not exist"
Oct 30 04:04:41.162444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162417    4528 scope.go:122] "RemoveContainer" containerID="2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"
Oct 30 04:04:41.162653 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162634    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c"} err="failed to get container status \"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c\": rpc error: code = NotFound desc = could not find container \"2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c\": container with ID starting with 2958c6eeae86d0c1809307863bf20897b860e24ab0ca3ba36250ddb538e65a4c not found: ID does not exist"
Oct 30 04:04:41.162700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162653    4528 scope.go:122] "RemoveContainer" containerID="8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"
Oct 30 04:04:41.162892 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162866    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5"} err="failed to get container status \"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5\": rpc error: code = NotFound desc = could not find container \"8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5\": container with ID starting with 8ad42ecea848192a74ed2f64110b079b3d6b0cc31d83f8444244e740b1e5dbc5 not found: ID does not exist"
Oct 30 04:04:41.162892 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.162890    4528 scope.go:122] "RemoveContainer" containerID="0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"
Oct 30 04:04:41.172178 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.172161    4528 scope.go:122] "RemoveContainer" containerID="20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"
Oct 30 04:04:41.180930 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.180915    4528 scope.go:122] "RemoveContainer" containerID="0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"
Oct 30 04:04:41.181254 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:41.181230    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8\": container with ID starting with 0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8 not found: ID does not exist" containerID="0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"
Oct 30 04:04:41.181323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181262    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"} err="failed to get container status \"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8\": rpc error: code = NotFound desc = could not find container \"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8\": container with ID starting with 0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8 not found: ID does not exist"
Oct 30 04:04:41.181323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181284    4528 scope.go:122] "RemoveContainer" containerID="20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"
Oct 30 04:04:41.181565 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:41.181546    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c\": container with ID starting with 20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c not found: ID does not exist" containerID="20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"
Oct 30 04:04:41.181617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181569    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"} err="failed to get container status \"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c\": rpc error: code = NotFound desc = could not find container \"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c\": container with ID starting with 20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c not found: ID does not exist"
Oct 30 04:04:41.181617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181583    4528 scope.go:122] "RemoveContainer" containerID="0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"
Oct 30 04:04:41.181813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181796    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8"} err="failed to get container status \"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8\": rpc error: code = NotFound desc = could not find container \"0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8\": container with ID starting with 0cce0abeeda1023d57e9d52b37eb3ebfc1eda6b687c079e3ea1558f5d8e14af8 not found: ID does not exist"
Oct 30 04:04:41.181813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.181826    4528 scope.go:122] "RemoveContainer" containerID="20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"
Oct 30 04:04:41.182059 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.182038    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c"} err="failed to get container status \"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c\": rpc error: code = NotFound desc = could not find container \"20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c\": container with ID starting with 20493e7bc7fe6eb891466c95003fd91fab19899cd36db7aff33fd38d5515ad1c not found: ID does not exist"
Oct 30 04:04:41.193616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.193595    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:04:41.198000 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.197979    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-8ff6v"]
Oct 30 04:04:41.230520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.230493    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:04:41.234409 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.234381    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-hd7n8"]
Oct 30 04:04:41.403996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.403943    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:41.530857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.530827    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:41.640677 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.640641    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:41.892170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.892132    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" path="/var/lib/kubelet/pods/941da6d5-0d51-4605-bb24-788f7d04065a/volumes"
Oct 30 04:04:41.892555 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:41.892542    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" path="/var/lib/kubelet/pods/987143a2-8d43-4e69-bfd8-e3b3ca7b60e5/volumes"
Oct 30 04:04:42.385906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:42.385871    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:42.518352 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:42.518319    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:42.636206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:42.636143    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:43.317878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.317848    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:04:43.388418 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388535    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388587    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388614    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388645    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388693    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388724    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388771    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388800    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388906    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388942    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.388977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.388983    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"3b105965-1079-4f5e-9f5a-b4f2952996e7\" (UID: \"3b105965-1079-4f5e-9f5a-b4f2952996e7\") "
Oct 30 04:04:43.389970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.389039    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:43.389970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.389272    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.390086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.390052    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:43.391795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.391752    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.391946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.391811    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:04:43.392400 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.392368    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "kube-api-access-czvts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:43.392941 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.392915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.393059 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.392968    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.393251 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.393216    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.393691 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.393460    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:04:43.394420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.394392    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:04:43.394862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.394843    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.396082 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.396063    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:04:43.406610 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.406510    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config" pod "3b105965-1079-4f5e-9f5a-b4f2952996e7" (UID: "3b105965-1079-4f5e-9f5a-b4f2952996e7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:04:43.490067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490033    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490064    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490077    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490088    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490119    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490130    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490139    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490148    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b105965-1079-4f5e-9f5a-b4f2952996e7-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490156    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490164    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czvts\" (UniqueName: \"kubernetes.io/projected/3b105965-1079-4f5e-9f5a-b4f2952996e7-kube-api-access-czvts\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490173    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b105965-1079-4f5e-9f5a-b4f2952996e7-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.490257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.490181    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b105965-1079-4f5e-9f5a-b4f2952996e7-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:04:43.500663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.500645    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:04:43.591515 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:43.591479    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:04:44.168417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.168332    4528 generic.go:356] "Generic (PLEG): container finished" podID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerID="355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5" exitCode=0
Oct 30 04:04:44.168417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.168396    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:04:44.168604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.168413    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerDied","Data":"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"}
Oct 30 04:04:44.168604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.168481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"3b105965-1079-4f5e-9f5a-b4f2952996e7","Type":"ContainerDied","Data":"cd5477a2cf4d1786fc7ebbe61ed6d9cf6833c7ad17e070645f032b287cfe0559"}
Oct 30 04:04:44.168604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.168514    4528 scope.go:122] "RemoveContainer" containerID="4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"
Oct 30 04:04:44.176527 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.176506    4528 scope.go:122] "RemoveContainer" containerID="965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"
Oct 30 04:04:44.184128 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.184112    4528 scope.go:122] "RemoveContainer" containerID="355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"
Oct 30 04:04:44.191484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.191466    4528 scope.go:122] "RemoveContainer" containerID="fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"
Oct 30 04:04:44.198841 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.198804    4528 scope.go:122] "RemoveContainer" containerID="4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"
Oct 30 04:04:44.199127 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:44.199107    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74\": container with ID starting with 4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74 not found: ID does not exist" containerID="4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"
Oct 30 04:04:44.199183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199135    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74"} err="failed to get container status \"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74\": rpc error: code = NotFound desc = could not find container \"4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74\": container with ID starting with 4a62104f41ec8d59f583775740ca8a8e9ef96b77fb9febf9bac2fb15c03f9a74 not found: ID does not exist"
Oct 30 04:04:44.199183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199153    4528 scope.go:122] "RemoveContainer" containerID="965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"
Oct 30 04:04:44.199394 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:44.199374    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548\": container with ID starting with 965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548 not found: ID does not exist" containerID="965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"
Oct 30 04:04:44.199458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199403    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548"} err="failed to get container status \"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548\": rpc error: code = NotFound desc = could not find container \"965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548\": container with ID starting with 965760284a87ec7a8f15d384795dbace5b2dbed43ac1ea285bd5733c762ea548 not found: ID does not exist"
Oct 30 04:04:44.199458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199426    4528 scope.go:122] "RemoveContainer" containerID="355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"
Oct 30 04:04:44.199680 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:44.199661    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5\": container with ID starting with 355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5 not found: ID does not exist" containerID="355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"
Oct 30 04:04:44.199744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199684    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5"} err="failed to get container status \"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5\": rpc error: code = NotFound desc = could not find container \"355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5\": container with ID starting with 355c3400c0f24a3669eeefe8ea51ccbb02ac669c285e1e5287bf0f55ee7a27f5 not found: ID does not exist"
Oct 30 04:04:44.199744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199699    4528 scope.go:122] "RemoveContainer" containerID="fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"
Oct 30 04:04:44.199926 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:04:44.199906    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c\": container with ID starting with fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c not found: ID does not exist" containerID="fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"
Oct 30 04:04:44.199972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.199932    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c"} err="failed to get container status \"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c\": rpc error: code = NotFound desc = could not find container \"fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c\": container with ID starting with fa8a24ef7276ad790f9f0890e8aaa23cc612ac785aba1bcc3fe53a36a8ae795c not found: ID does not exist"
Oct 30 04:04:44.231429 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.231400    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:04:44.242933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:44.242909    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:04:45.891870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:45.891830    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" path="/var/lib/kubelet/pods/3b105965-1079-4f5e-9f5a-b4f2952996e7/volumes"
Oct 30 04:04:50.764157 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:50.764125    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:50.897786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:50.897742    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.020343 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.020259    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.092258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092228    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:04:51.092563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092546    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="prometheus"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092564    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="prometheus"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092582    4528 cpu_manager.go:414] "containerMap: removing container" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="kube-rbac-proxy"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092589    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="kube-rbac-proxy"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092609    4528 cpu_manager.go:414] "containerMap: removing container" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="prom-label-proxy"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092616    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="prom-label-proxy"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092626    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092633    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo"
Oct 30 04:04:51.092647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092645    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="thanos-sidecar"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092652    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="thanos-sidecar"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092665    4528 cpu_manager.go:414] "containerMap: removing container" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="kube-rbac-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092674    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="kube-rbac-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092685    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092692    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092707    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="init-config-reloader"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092715    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="init-config-reloader"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="config-reloader"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092733    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="config-reloader"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092745    4528 cpu_manager.go:414] "containerMap: removing container" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="prom-label-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092752    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="prom-label-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092771    4528 cpu_manager.go:414] "containerMap: removing container" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway-opa"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092778    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway-opa"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092873    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="config-reloader"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092885    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="prom-label-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092896    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway-opa"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092905    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="prometheus"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092914    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="987143a2-8d43-4e69-bfd8-e3b3ca7b60e5" containerName="kube-rbac-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092925    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="prom-label-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092932    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092941    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b105965-1079-4f5e-9f5a-b4f2952996e7" containerName="thanos-sidecar"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092950    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="941da6d5-0d51-4605-bb24-788f7d04065a" containerName="kube-rbac-proxy"
Oct 30 04:04:51.093080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.092960    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3741d9-c061-4a50-9038-7f9bc63634d5" containerName="tempo-gateway"
Oct 30 04:04:51.097908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.097888    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.116701 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.116677    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:04:51.137911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.137879    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.139340 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.139313    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:04:51.142970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.142952    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.162920 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.162884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:04:51.177013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.176968    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:04:51.177136 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177057    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\""
Oct 30 04:04:51.177136 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177072    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\""
Oct 30 04:04:51.177255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177159    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-5vnnw\""
Oct 30 04:04:51.177255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177217    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\""
Oct 30 04:04:51.177255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177244    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\""
Oct 30 04:04:51.177386 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177275    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:04:51.177386 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177333    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:04:51.177386 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177334    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\""
Oct 30 04:04:51.177531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.177425    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-n4ltn\""
Oct 30 04:04:51.252468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252429    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5n9v\" (UniqueName: \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.252468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252472    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.252688 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252560    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.252688 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252611    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.252808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252717    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqn98\" (UniqueName: \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.252808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.252808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252797    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.252960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.252853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353276    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353325    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqn98\" (UniqueName: \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353489    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353526    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.353589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353573    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5n9v\" (UniqueName: \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.353841 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.353841 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.353645    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.354255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.354224    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.354394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.354303    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.354394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.354355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.354394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.354369    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.355890 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.355871    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.356015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.355999    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.379245 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.379213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5n9v\" (UniqueName: \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\") pod \"data-science-thanos-querier-proxy-58d8d65bf9-4p8mz\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.379346 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.379301    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqn98\" (UniqueName: \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\") pod \"data-science-prometheus-namespace-proxy-5879977c4b-b7whp\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.390511 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.390485    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.412459 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.412438    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:04:51.456687 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.456657    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:04:51.726951 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.726884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:04:51.729700 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:04:51.729671    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da2a4b1_daaa_483f_9c1a_2f5efc43909f.slice/crio-a7214b168c307b354af808dcd330d3a0adf34ad26348d1056776199306630916 WatchSource:0}: Error finding container a7214b168c307b354af808dcd330d3a0adf34ad26348d1056776199306630916: Status 404 returned error can't find the container with id a7214b168c307b354af808dcd330d3a0adf34ad26348d1056776199306630916
Oct 30 04:04:51.748510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.748479    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.766237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.766219    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:04:51.769073 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:04:51.769048    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1884868_92ae_4265_a51c_fa7e57cc79dd.slice/crio-28454b7e14c6abdcc36f9da0bb1a1e6f924e5b7cd0071735b74b971a74fd66b6 WatchSource:0}: Error finding container 28454b7e14c6abdcc36f9da0bb1a1e6f924e5b7cd0071735b74b971a74fd66b6: Status 404 returned error can't find the container with id 28454b7e14c6abdcc36f9da0bb1a1e6f924e5b7cd0071735b74b971a74fd66b6
Oct 30 04:04:51.868097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.867424    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:51.997455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:51.997367    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:52.204253 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.204216    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerStarted","Data":"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"}
Oct 30 04:04:52.204474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.204275    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerStarted","Data":"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"}
Oct 30 04:04:52.204474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.204305    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerStarted","Data":"28454b7e14c6abdcc36f9da0bb1a1e6f924e5b7cd0071735b74b971a74fd66b6"}
Oct 30 04:04:52.205930 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.205905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerStarted","Data":"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"}
Oct 30 04:04:52.206030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.205952    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerStarted","Data":"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"}
Oct 30 04:04:52.206030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.205975    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerStarted","Data":"a7214b168c307b354af808dcd330d3a0adf34ad26348d1056776199306630916"}
Oct 30 04:04:52.246788 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.246735    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" podStartSLOduration=1.246722565 podStartE2EDuration="1.246722565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.245927083 +0000 UTC m=+5050.470176361" watchObservedRunningTime="2026-10-30 04:04:52.246722565 +0000 UTC m=+5050.470971836"
Oct 30 04:04:52.287170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.287064    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" podStartSLOduration=1.287045405 podStartE2EDuration="1.287045405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.286738821 +0000 UTC m=+5050.510988318" watchObservedRunningTime="2026-10-30 04:04:52.287045405 +0000 UTC m=+5050.511294683"
Oct 30 04:04:52.787777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.787736    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:52.932807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:52.932772    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:55.510647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.510611    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:04:55.513922 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.513904    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:04:55.528679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.528661    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:04:55.541886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.541859    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:04:55.591454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.591397    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5qg\" (UniqueName: \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb\" (UID: \"ae0a4c6c-f4ed-4859-b442-70c3240a95dc\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:04:55.691784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.691753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5qg\" (UniqueName: \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb\" (UID: \"ae0a4c6c-f4ed-4859-b442-70c3240a95dc\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:04:55.707648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.707624    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:55.714343 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.714320    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5qg\" (UniqueName: \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb\" (UID: \"ae0a4c6c-f4ed-4859-b442-70c3240a95dc\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:04:55.825764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.825681    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:04:55.949327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:55.949282    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:56.062003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.061964    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:56.119093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.119017    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:04:56.121772 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:04:56.121743    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae0a4c6c_f4ed_4859_b442_70c3240a95dc.slice/crio-ddf084f66e513c3582798af8a01ea785cd8a8d9304c37d61068c2e4aca3e09ba WatchSource:0}: Error finding container ddf084f66e513c3582798af8a01ea785cd8a8d9304c37d61068c2e4aca3e09ba: Status 404 returned error can't find the container with id ddf084f66e513c3582798af8a01ea785cd8a8d9304c37d61068c2e4aca3e09ba
Oct 30 04:04:56.175134 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.175105    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:56.228069 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.228030    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" event={"ID":"ae0a4c6c-f4ed-4859-b442-70c3240a95dc","Type":"ContainerStarted","Data":"0d1cf0513f3228b1ae09d78814e7f72001c59014034e42f9d11bb5ae460a26ef"}
Oct 30 04:04:56.228240 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.228086    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" event={"ID":"ae0a4c6c-f4ed-4859-b442-70c3240a95dc","Type":"ContainerStarted","Data":"ddf084f66e513c3582798af8a01ea785cd8a8d9304c37d61068c2e4aca3e09ba"}
Oct 30 04:04:56.265348 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.265067    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" podStartSLOduration=1.265050107 podStartE2EDuration="1.265050107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:56.264841176 +0000 UTC m=+5054.489090451" watchObservedRunningTime="2026-10-30 04:04:56.265050107 +0000 UTC m=+5054.489299386"
Oct 30 04:04:56.497306 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.497271    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:56.538314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.538279    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:04:56.542120 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.542092    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.569968 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.569940    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:04:56.595915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.595879    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:04:56.596037 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.595981    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-s78sv\""
Oct 30 04:04:56.600705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.600677    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:04:56.600705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.600676    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:04:56.600886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.600678    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:04:56.600886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.600689    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:04:56.600886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.600700    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:04:56.625107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.625067    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:04:56.642679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.642657    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:04:56.642829 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.642801    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:04:56.642934 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.642921    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:04:56.697689 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697663    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697707    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697811    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697857    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697872    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.697971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.697952    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.698058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.698002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.698058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.698020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.698058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.698035    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzzx\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.698172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.698064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.698172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.698147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799080    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799113    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799139    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799252    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799267    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzzx\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799291    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799310    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799404    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.799875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.799467    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.800575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.800222    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.800575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.800380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.801209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.801179    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.802312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.802248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.802444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.802419    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.802521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.802489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.802588 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.802571    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.802640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.802605    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.803102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.803081    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.803263 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.803242    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.803636 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.803612    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:04:56.823107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:04:56.823087    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzzx\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.329184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.329145    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.356995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.356960    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.430463 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.430421    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.430625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.430566    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.444447 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.444415    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") DevicePath \"csi-e8943ccbf9e44c5d22a63e7297537a58134696922d8a2a16f63173089ab7966e\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.545454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.545421    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a535375d22e374ba333968465471c7604a06f9d95a513910fe4bb083b629e986/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.570658 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.570634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:00.762268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:00.762223    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:01.095746 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:05:01.095715    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a415922_b74a_4f0b_bd27_4c41df87a63d.slice/crio-d9da013aefa53017bc01b3d561e166a8a2c6b2123f40e442a644fc385d338af8 WatchSource:0}: Error finding container d9da013aefa53017bc01b3d561e166a8a2c6b2123f40e442a644fc385d338af8: Status 404 returned error can't find the container with id d9da013aefa53017bc01b3d561e166a8a2c6b2123f40e442a644fc385d338af8
Oct 30 04:05:01.104564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:01.104541    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:05:01.256283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:01.256246    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerStarted","Data":"d9da013aefa53017bc01b3d561e166a8a2c6b2123f40e442a644fc385d338af8"}
Oct 30 04:05:02.261439 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:02.261403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerStarted","Data":"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"}
Oct 30 04:05:04.630454 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:05:04.630417    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:05:05.274553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:05.274518    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerID="e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8" exitCode=0
Oct 30 04:05:05.274741 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:05.274591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerDied","Data":"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"}
Oct 30 04:05:06.287895 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:06.287857    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerStarted","Data":"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"}
Oct 30 04:05:06.288288 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:06.287949    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerStarted","Data":"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"}
Oct 30 04:05:06.288288 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:06.287974    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerStarted","Data":"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"}
Oct 30 04:05:06.362677 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:06.362609    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=6.877231453 podStartE2EDuration="10.362597123s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.48536567s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:06.350397925 +0000 UTC m=+5064.574647202" watchObservedRunningTime="2026-10-30 04:05:06.362597123 +0000 UTC m=+5064.586846394"
Oct 30 04:05:06.935513 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:06.935477    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:08.153057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:08.153021    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:10.762362 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:10.762318    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:12.675314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:12.675273    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:12.791201 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:12.791155    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:13.299198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:13.299159    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:13.420277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:13.420236    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:13.631007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:13.630976    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:15.763110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:15.763079    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:15.768498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:15.768474    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:16.333976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:16.333946    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:05:18.407953 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:18.407916    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:18.526055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:18.526015    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:19.784160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:19.784124    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:19.891487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:19.891445    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:20.099999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:20.099906    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.069939 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:24.069859    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.191152 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:24.191117    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.311563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:24.311527    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.425115 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:24.425082    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:24.640916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:24.640871    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:29.562901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:29.562866    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:29.687267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:29.687226    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:30.794421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:30.794382    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:30.913245 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:30.913214    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:31.029740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:31.029703    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:31.248051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:31.248019    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:34.631111 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:05:34.631076    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:05:35.320620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:35.320580    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:35.440540 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:35.440505    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:37.406084 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:37.406049    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:37.528692 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:37.528618    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:37.643756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:37.643719    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:40.116780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:40.116739    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:40.946202 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:40.946164    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:41.062727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:41.062693    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:41.763346 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:41.763311    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:41.882201 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:41.882166    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:42.002192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:42.002155    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:42.618231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:42.618200    4528 scope.go:122] "RemoveContainer" containerID="1339c97d686204519f3df975f512177e8e75040dd30503ebd1b1e7d5071fbb2f"
Oct 30 04:05:46.640230 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:46.640188    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:46.750233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:46.750200    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:48.135172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:48.135133    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:48.252960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:48.252929    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:48.367689 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:48.367655    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:52.343224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:52.343187    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:52.459999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:52.459960    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:52.574429 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:52.574395    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:52.717950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:52.717917    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:58.039297 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:58.039247    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:58.168692 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:58.168657    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:05:58.605800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:05:58.605765    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:00.901141 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.901103    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:06:00.901613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.901331    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="kube-rbac-proxy" containerID="cri-o://e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b" gracePeriod=30
Oct 30 04:06:00.901613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.901443    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="prom-label-proxy" containerID="cri-o://f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d" gracePeriod=30
Oct 30 04:06:00.918408 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.918381    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:06:00.918646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.918611    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="kube-rbac-proxy" containerID="cri-o://a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0" gracePeriod=30
Oct 30 04:06:00.918646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:00.918616    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="prom-label-proxy" containerID="cri-o://6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a" gracePeriod=30
Oct 30 04:06:01.158960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.158898    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:06:01.179019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.178992    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:06:01.220198 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220157    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\" (UniqueName: \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\") pod \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") "
Oct 30 04:06:01.220385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220252    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\" (UniqueName: \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\") pod \"a1884868-92ae-4265-a51c-fa7e57cc79dd\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") "
Oct 30 04:06:01.220385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220280    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\") pod \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") "
Oct 30 04:06:01.220385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220319    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\" (UniqueName: \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\") pod \"a1884868-92ae-4265-a51c-fa7e57cc79dd\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") "
Oct 30 04:06:01.220385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220345    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\") pod \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") "
Oct 30 04:06:01.220623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\" (UniqueName: \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\") pod \"a1884868-92ae-4265-a51c-fa7e57cc79dd\" (UID: \"a1884868-92ae-4265-a51c-fa7e57cc79dd\") "
Oct 30 04:06:01.220623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220420    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\") pod \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") "
Oct 30 04:06:01.220623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220456    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\" (UniqueName: \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\") pod \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\" (UID: \"5da2a4b1-daaa-483f-9c1a-2f5efc43909f\") "
Oct 30 04:06:01.220785 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220661    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config" pod "a1884868-92ae-4265-a51c-fa7e57cc79dd" (UID: "a1884868-92ae-4265-a51c-fa7e57cc79dd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.220785 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.220675    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config" pod "5da2a4b1-daaa-483f-9c1a-2f5efc43909f" (UID: "5da2a4b1-daaa-483f-9c1a-2f5efc43909f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.221655 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.221100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca" pod "5da2a4b1-daaa-483f-9c1a-2f5efc43909f" (UID: "5da2a4b1-daaa-483f-9c1a-2f5efc43909f"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.221655 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.221100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca" pod "5da2a4b1-daaa-483f-9c1a-2f5efc43909f" (UID: "5da2a4b1-daaa-483f-9c1a-2f5efc43909f"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.224345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.224297    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls" pod "5da2a4b1-daaa-483f-9c1a-2f5efc43909f" (UID: "5da2a4b1-daaa-483f-9c1a-2f5efc43909f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.224345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.224308    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls" pod "a1884868-92ae-4265-a51c-fa7e57cc79dd" (UID: "a1884868-92ae-4265-a51c-fa7e57cc79dd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.226459 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.226431    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v" pod "a1884868-92ae-4265-a51c-fa7e57cc79dd" (UID: "a1884868-92ae-4265-a51c-fa7e57cc79dd"). InnerVolumeSpecName "kube-api-access-j5n9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:01.226571 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.226514    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98" pod "5da2a4b1-daaa-483f-9c1a-2f5efc43909f" (UID: "5da2a4b1-daaa-483f-9c1a-2f5efc43909f"). InnerVolumeSpecName "kube-api-access-gqn98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:01.241585 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.241553    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:06:01.241805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.241769    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" podUID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" containerName="thanos-querier" containerID="cri-o://0d1cf0513f3228b1ae09d78814e7f72001c59014034e42f9d11bb5ae460a26ef" gracePeriod=30
Oct 30 04:06:01.308597 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.308556    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:01.321074 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321040    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqn98\" (UniqueName: \"kubernetes.io/projected/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-kube-api-access-gqn98\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321074 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321075    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1884868-92ae-4265-a51c-fa7e57cc79dd-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321074 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321085    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321093    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1884868-92ae-4265-a51c-fa7e57cc79dd-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321101    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-prometheus-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321109    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5n9v\" (UniqueName: \"kubernetes.io/projected/a1884868-92ae-4265-a51c-fa7e57cc79dd-kube-api-access-j5n9v\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321117    4528 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-client-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.321331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.321126    4528 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5da2a4b1-daaa-483f-9c1a-2f5efc43909f-tls\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.392628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.392593    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:06:01.393517 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.393484    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="thanos-sidecar" containerID="cri-o://08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc" gracePeriod=600
Oct 30 04:06:01.393896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.393863    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="config-reloader" containerID="cri-o://90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944" gracePeriod=600
Oct 30 04:06:01.394023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.393999    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="prometheus" containerID="cri-o://4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea" gracePeriod=600
Oct 30 04:06:01.425932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.425900    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:01.524199 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524166    4528 generic.go:356] "Generic (PLEG): container finished" podID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerID="f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d" exitCode=0
Oct 30 04:06:01.524199 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524196    4528 generic.go:356] "Generic (PLEG): container finished" podID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerID="e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b" exitCode=0
Oct 30 04:06:01.524199 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524195    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerDied","Data":"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"}
Oct 30 04:06:01.524443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524217    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"
Oct 30 04:06:01.524443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524256    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerDied","Data":"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"}
Oct 30 04:06:01.524443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524278    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp" event={"ID":"5da2a4b1-daaa-483f-9c1a-2f5efc43909f","Type":"ContainerDied","Data":"a7214b168c307b354af808dcd330d3a0adf34ad26348d1056776199306630916"}
Oct 30 04:06:01.524443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.524324    4528 scope.go:122] "RemoveContainer" containerID="f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"
Oct 30 04:06:01.525985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.525960    4528 generic.go:356] "Generic (PLEG): container finished" podID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerID="6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a" exitCode=0
Oct 30 04:06:01.525985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.525984    4528 generic.go:356] "Generic (PLEG): container finished" podID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerID="a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0" exitCode=0
Oct 30 04:06:01.526165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.526009    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"
Oct 30 04:06:01.526165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.526019    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerDied","Data":"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"}
Oct 30 04:06:01.526165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.526082    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerDied","Data":"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"}
Oct 30 04:06:01.526165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.526116    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz" event={"ID":"a1884868-92ae-4265-a51c-fa7e57cc79dd","Type":"ContainerDied","Data":"28454b7e14c6abdcc36f9da0bb1a1e6f924e5b7cd0071735b74b971a74fd66b6"}
Oct 30 04:06:01.527741 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.527720    4528 generic.go:356] "Generic (PLEG): container finished" podID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" containerID="0d1cf0513f3228b1ae09d78814e7f72001c59014034e42f9d11bb5ae460a26ef" exitCode=0
Oct 30 04:06:01.527868 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.527795    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" event={"ID":"ae0a4c6c-f4ed-4859-b442-70c3240a95dc","Type":"ContainerDied","Data":"0d1cf0513f3228b1ae09d78814e7f72001c59014034e42f9d11bb5ae460a26ef"}
Oct 30 04:06:01.532287 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.532263    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerID="08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc" exitCode=0
Oct 30 04:06:01.532287 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.532287    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerID="4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea" exitCode=0
Oct 30 04:06:01.532481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.532325    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerDied","Data":"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"}
Oct 30 04:06:01.532481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.532406    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerDied","Data":"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"}
Oct 30 04:06:01.539710 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.539688    4528 scope.go:122] "RemoveContainer" containerID="e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"
Oct 30 04:06:01.544069 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.544043    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:01.580923 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.580901    4528 scope.go:122] "RemoveContainer" containerID="f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"
Oct 30 04:06:01.581296 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:01.581273    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d\": container with ID starting with f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d not found: ID does not exist" containerID="f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"
Oct 30 04:06:01.581386 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581305    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"} err="failed to get container status \"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d\": rpc error: code = NotFound desc = could not find container \"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d\": container with ID starting with f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d not found: ID does not exist"
Oct 30 04:06:01.581386 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581325    4528 scope.go:122] "RemoveContainer" containerID="e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"
Oct 30 04:06:01.581613 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:01.581582    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b\": container with ID starting with e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b not found: ID does not exist" containerID="e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"
Oct 30 04:06:01.581652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581626    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"} err="failed to get container status \"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b\": rpc error: code = NotFound desc = could not find container \"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b\": container with ID starting with e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b not found: ID does not exist"
Oct 30 04:06:01.581652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581648    4528 scope.go:122] "RemoveContainer" containerID="f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"
Oct 30 04:06:01.581952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581923    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d"} err="failed to get container status \"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d\": rpc error: code = NotFound desc = could not find container \"f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d\": container with ID starting with f743bb077598cd01daa7075ca64cb5379be9176f7c0c8ee0e7cdde1d4140825d not found: ID does not exist"
Oct 30 04:06:01.581952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.581952    4528 scope.go:122] "RemoveContainer" containerID="e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"
Oct 30 04:06:01.582265 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.582238    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b"} err="failed to get container status \"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b\": rpc error: code = NotFound desc = could not find container \"e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b\": container with ID starting with e8885bdc98bcbf3d96a842d09780590410d7545cdf7ecdacd9fac6a463a59e9b not found: ID does not exist"
Oct 30 04:06:01.582316 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.582269    4528 scope.go:122] "RemoveContainer" containerID="6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"
Oct 30 04:06:01.593965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.593943    4528 scope.go:122] "RemoveContainer" containerID="a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"
Oct 30 04:06:01.595858 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.595834    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:06:01.602689 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.602662    4528 scope.go:122] "RemoveContainer" containerID="6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"
Oct 30 04:06:01.603092 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:01.603070    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a\": container with ID starting with 6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a not found: ID does not exist" containerID="6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"
Oct 30 04:06:01.603204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603105    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"} err="failed to get container status \"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a\": rpc error: code = NotFound desc = could not find container \"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a\": container with ID starting with 6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a not found: ID does not exist"
Oct 30 04:06:01.603204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603130    4528 scope.go:122] "RemoveContainer" containerID="a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"
Oct 30 04:06:01.603509 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:01.603487    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0\": container with ID starting with a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0 not found: ID does not exist" containerID="a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"
Oct 30 04:06:01.603565 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603523    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"} err="failed to get container status \"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0\": rpc error: code = NotFound desc = could not find container \"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0\": container with ID starting with a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0 not found: ID does not exist"
Oct 30 04:06:01.603565 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603542    4528 scope.go:122] "RemoveContainer" containerID="6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"
Oct 30 04:06:01.603869 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603842    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a"} err="failed to get container status \"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a\": rpc error: code = NotFound desc = could not find container \"6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a\": container with ID starting with 6653c02920f4e9c9ee7a0c8fa2536272ef1a77ae603a22fef80a0a8f5557b30a not found: ID does not exist"
Oct 30 04:06:01.603977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.603870    4528 scope.go:122] "RemoveContainer" containerID="a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"
Oct 30 04:06:01.604164 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.604142    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0"} err="failed to get container status \"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0\": rpc error: code = NotFound desc = could not find container \"a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0\": container with ID starting with a6bf3d1d480efc87c3c00830c2e72eb6cc76b339d2c03345f009ae1daf9452d0 not found: ID does not exist"
Oct 30 04:06:01.623274 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.623248    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\" (UniqueName: \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\") pod \"ae0a4c6c-f4ed-4859-b442-70c3240a95dc\" (UID: \"ae0a4c6c-f4ed-4859-b442-70c3240a95dc\") "
Oct 30 04:06:01.625307 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.625284    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg" pod "ae0a4c6c-f4ed-4859-b442-70c3240a95dc" (UID: "ae0a4c6c-f4ed-4859-b442-70c3240a95dc"). InnerVolumeSpecName "kube-api-access-xh5qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:01.629609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.629583    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:06:01.634216 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.634188    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-58d8d65bf9-4p8mz"]
Oct 30 04:06:01.668094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.668039    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:06:01.672292 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.672266    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5879977c4b-b7whp"]
Oct 30 04:06:01.675599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.675576    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:06:01.724480 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724451    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.724672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724506    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.724672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724543    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.724672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724593    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.724672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724621    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.724672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724650    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724676    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724721    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724833    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724876    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724905    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.724915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.725050    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.725165    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5qg\" (UniqueName: \"kubernetes.io/projected/ae0a4c6c-f4ed-4859-b442-70c3240a95dc-kube-api-access-xh5qg\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.725190    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.725334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.725207    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.727739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.727710    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:01.727861 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.727790    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.728515 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.728475    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.728662 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.728512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.728759 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.728718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "kube-api-access-wbzzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:01.730032 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.728985    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.730032 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.729949    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.730677 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.730656    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:06:01.739026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.739002    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:01.825870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.825829    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.826063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.825882    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\") pod \"4a415922-b74a-4f0b-bd27-4c41df87a63d\" (UID: \"4a415922-b74a-4f0b-bd27-4c41df87a63d\") "
Oct 30 04:06:01.826063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826053    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzzx\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-kube-api-access-wbzzx\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826077    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826089    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826099    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a415922-b74a-4f0b-bd27-4c41df87a63d-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826115    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826129    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826138    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826147    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826159    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4a415922-b74a-4f0b-bd27-4c41df87a63d-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.826551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.826188    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:01.828125 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.828094    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out" pod "4a415922-b74a-4f0b-bd27-4c41df87a63d" (UID: "4a415922-b74a-4f0b-bd27-4c41df87a63d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:01.836960 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.836940    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-eb589c5c-5356-4187-b8d6-3245b2940faa" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:06:01.892758 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.892728    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" path="/var/lib/kubelet/pods/5da2a4b1-daaa-483f-9c1a-2f5efc43909f/volumes"
Oct 30 04:06:01.893215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.893200    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" path="/var/lib/kubelet/pods/a1884868-92ae-4265-a51c-fa7e57cc79dd/volumes"
Oct 30 04:06:01.927478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.927413    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a415922-b74a-4f0b-bd27-4c41df87a63d-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.927478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.927441    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a415922-b74a-4f0b-bd27-4c41df87a63d-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:06:01.927478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:01.927455    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-eb589c5c-5356-4187-b8d6-3245b2940faa\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0155337cccbe571a4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-e8943ccbf9e44c5d22a63e7297537a58134696922d8a2a16f63173089ab7966e\""
Oct 30 04:06:02.130396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.130358    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:02.247370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.247275    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:02.360020 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.359991    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:02.545498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.545414    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb" event={"ID":"ae0a4c6c-f4ed-4859-b442-70c3240a95dc","Type":"ContainerDied","Data":"ddf084f66e513c3582798af8a01ea785cd8a8d9304c37d61068c2e4aca3e09ba"}
Oct 30 04:06:02.545498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.545470    4528 scope.go:122] "RemoveContainer" containerID="0d1cf0513f3228b1ae09d78814e7f72001c59014034e42f9d11bb5ae460a26ef"
Oct 30 04:06:02.545498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.545424    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"
Oct 30 04:06:02.548434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.548404    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerID="90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944" exitCode=0
Oct 30 04:06:02.548559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.548496    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerDied","Data":"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"}
Oct 30 04:06:02.548559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.548527    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"
Oct 30 04:06:02.548646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.548591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"4a415922-b74a-4f0b-bd27-4c41df87a63d","Type":"ContainerDied","Data":"d9da013aefa53017bc01b3d561e166a8a2c6b2123f40e442a644fc385d338af8"}
Oct 30 04:06:02.555306 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.555284    4528 scope.go:122] "RemoveContainer" containerID="08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"
Oct 30 04:06:02.562756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.562738    4528 scope.go:122] "RemoveContainer" containerID="90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"
Oct 30 04:06:02.570410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.570391    4528 scope.go:122] "RemoveContainer" containerID="4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"
Oct 30 04:06:02.578232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.578209    4528 scope.go:122] "RemoveContainer" containerID="e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"
Oct 30 04:06:02.585837 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.585806    4528 scope.go:122] "RemoveContainer" containerID="08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"
Oct 30 04:06:02.586137 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:02.586116    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc\": container with ID starting with 08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc not found: ID does not exist" containerID="08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"
Oct 30 04:06:02.586185 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586146    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc"} err="failed to get container status \"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc\": rpc error: code = NotFound desc = could not find container \"08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc\": container with ID starting with 08e01b0421e87b8d7f4850fcbc51f4ad4e3cef32d7bc5c9c56fa903cd5dce0fc not found: ID does not exist"
Oct 30 04:06:02.586185 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586165    4528 scope.go:122] "RemoveContainer" containerID="90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"
Oct 30 04:06:02.586429 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:02.586410    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944\": container with ID starting with 90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944 not found: ID does not exist" containerID="90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"
Oct 30 04:06:02.586482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586434    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944"} err="failed to get container status \"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944\": rpc error: code = NotFound desc = could not find container \"90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944\": container with ID starting with 90b0fe3715bb48708093ef3b12d3bbcdb99ca7c1feb76822bd08f59532369944 not found: ID does not exist"
Oct 30 04:06:02.586482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586451    4528 scope.go:122] "RemoveContainer" containerID="4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"
Oct 30 04:06:02.586679 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:02.586663    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea\": container with ID starting with 4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea not found: ID does not exist" containerID="4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"
Oct 30 04:06:02.586730 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586692    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea"} err="failed to get container status \"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea\": rpc error: code = NotFound desc = could not find container \"4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea\": container with ID starting with 4bc81107b6385a197e4b6c6d767aefeb6c42f463143d2e61de840549570beeea not found: ID does not exist"
Oct 30 04:06:02.586730 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586707    4528 scope.go:122] "RemoveContainer" containerID="e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"
Oct 30 04:06:02.586971 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:02.586954    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8\": container with ID starting with e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8 not found: ID does not exist" containerID="e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"
Oct 30 04:06:02.587029 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.586974    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8"} err="failed to get container status \"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8\": rpc error: code = NotFound desc = could not find container \"e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8\": container with ID starting with e4b4ac9e26f9e91be4ea865561fac0b7a10fcae1825ff0045a18e2b3c78c92d8 not found: ID does not exist"
Oct 30 04:06:02.597028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.596998    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:06:02.608645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.608621    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"]
Oct 30 04:06:02.641949 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.641919    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:06:02.645916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:02.645881    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-r2tgb"]
Oct 30 04:06:03.123661 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:03.123627    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:03.247913 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:03.247856    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:03.372244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:03.372209    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:03.891418 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:03.891384    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" path="/var/lib/kubelet/pods/4a415922-b74a-4f0b-bd27-4c41df87a63d/volumes"
Oct 30 04:06:03.891857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:03.891844    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" path="/var/lib/kubelet/pods/ae0a4c6c-f4ed-4859-b442-70c3240a95dc/volumes"
Oct 30 04:06:04.252801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:04.252713    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:04.630642 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:04.630609    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:06:05.424245 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:05.424211    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.383461 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383428    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383684    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="kube-rbac-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383694    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="kube-rbac-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383702    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="thanos-sidecar"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383708    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="thanos-sidecar"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383723    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="prom-label-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383728    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="prom-label-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383737    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="kube-rbac-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383741    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="kube-rbac-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383748    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="init-config-reloader"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383752    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="init-config-reloader"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383759    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="prom-label-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383763    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="prom-label-proxy"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383770    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="config-reloader"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383774    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="config-reloader"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383783    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" containerName="thanos-querier"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383787    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" containerName="thanos-querier"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383794    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="prometheus"
Oct 30 04:06:24.383850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383798    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="prometheus"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383919    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="kube-rbac-proxy"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383928    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="thanos-sidecar"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383937    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1884868-92ae-4265-a51c-fa7e57cc79dd" containerName="prom-label-proxy"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383943    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="kube-rbac-proxy"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383949    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da2a4b1-daaa-483f-9c1a-2f5efc43909f" containerName="prom-label-proxy"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383954    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="config-reloader"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383967    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0a4c6c-f4ed-4859-b442-70c3240a95dc" containerName="thanos-querier"
Oct 30 04:06:24.384377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.383972    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a415922-b74a-4f0b-bd27-4c41df87a63d" containerName="prometheus"
Oct 30 04:06:24.388611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.388584    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.412578 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.412548    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.415341 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.415318    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:06:24.475777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.475736    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:06:24.475777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.475744    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:06:24.475777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.475744    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 04:06:24.476023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.475917    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-qxnzq\""
Oct 30 04:06:24.476023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.475981    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 04:06:24.476095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.476042    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\""
Oct 30 04:06:24.476095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.476071    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\""
Oct 30 04:06:24.499377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.499353    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.499492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.499390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.499492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.499415    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.499492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.499459    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msg7m\" (UniqueName: \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.499616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.499514    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.532687 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.532653    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.600390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.600356    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.600557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.600395    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.600557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.600419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msg7m\" (UniqueName: \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.600557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.600458    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.600557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.600515    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.601223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.601198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.601352 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.601248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.602808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.602788    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.602899 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.602848    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.626304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.626280    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msg7m\" (UniqueName: \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\") pod \"data-science-lokistack-distributor-58947679b7-v9w9m\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.647116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.647058    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:06:24.651257 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.651239    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.655906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.655879    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.674494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.674473    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:06:24.700249 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.700222    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:24.701466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701445    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.701527 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701491    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.701625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701602    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.701752 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701638    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhrl\" (UniqueName: \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.701814 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.701917 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.701885    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.742620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.742586    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-shared-secret\""
Oct 30 04:06:24.742811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.742795    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\""
Oct 30 04:06:24.742811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.742804    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\""
Oct 30 04:06:24.789258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.787584    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.803166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.803166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803168    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.803387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803211    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.803387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803261    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.803387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803296    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhrl\" (UniqueName: \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.803387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.803351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.804056 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.804029    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.804212 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.804162    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.805936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.805910    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.806049 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.805917    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.806100 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.806081    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.827601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.827577    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhrl\" (UniqueName: \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\") pod \"data-science-lokistack-querier-84985db556-tx4kz\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.905083 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.904691    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:24.963519 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.963489    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:24.988632 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.988594    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:06:24.993732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:24.993704    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.000985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.000958    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:06:25.008555 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.008068    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:06:25.008948 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.008928    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 04:06:25.009459 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:06:25.009433    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1972c32_1497_43e9_9837_279d28a0b6ac.slice/crio-2e25c50995d2b98f59e16bba4aa927088bb410ffb91ef836b4daa89eda7d8ba6 WatchSource:0}: Error finding container 2e25c50995d2b98f59e16bba4aa927088bb410ffb91ef836b4daa89eda7d8ba6: Status 404 returned error can't find the container with id 2e25c50995d2b98f59e16bba4aa927088bb410ffb91ef836b4daa89eda7d8ba6
Oct 30 04:06:25.014264 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.011733    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:06:25.016796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.016409    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:06:25.017231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.017208    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:06:25.020959 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.020935    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:25.054204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.054170    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 04:06:25.054338 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.054291    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 04:06:25.105222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105185    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105223    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105370    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105417    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzgr\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105554 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105529    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.105586 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.105563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.142782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.142733    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:25.205983 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.205903    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.205983 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.205948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.205983 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.205973    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.205999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206019    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206033    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206050    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzgr\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206117    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.206888 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206857    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.207007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206869    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.207007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.206951    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.208681 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.208654    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.208793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.208683    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.209004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.208985    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.209077 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.209036    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.229553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.229533    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzgr\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\") pod \"data-science-lokistack-gateway-d7f6645-c4n28\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.255197 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.255168    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:25.277814 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.277784    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:06:25.280462 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:06:25.280435    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ed00fd_b3ff_46ce_a12d_c1c529f72cc2.slice/crio-9100aef4fe4c574837f7b664028d71399367e7a58f99c877f5628dc0055813ef WatchSource:0}: Error finding container 9100aef4fe4c574837f7b664028d71399367e7a58f99c877f5628dc0055813ef: Status 404 returned error can't find the container with id 9100aef4fe4c574837f7b664028d71399367e7a58f99c877f5628dc0055813ef
Oct 30 04:06:25.310917 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.310894    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:25.623319 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.623255    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:06:25.626395 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:06:25.626362    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ff0038_854c_4145_948e_5f794ba65fcd.slice/crio-5600b0b8634f4e74e9b25c8386610223dc6d5cd5b6bd6333565cafe38dbfcbe2 WatchSource:0}: Error finding container 5600b0b8634f4e74e9b25c8386610223dc6d5cd5b6bd6333565cafe38dbfcbe2: Status 404 returned error can't find the container with id 5600b0b8634f4e74e9b25c8386610223dc6d5cd5b6bd6333565cafe38dbfcbe2
Oct 30 04:06:25.648588 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.648558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" event={"ID":"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2","Type":"ContainerStarted","Data":"9100aef4fe4c574837f7b664028d71399367e7a58f99c877f5628dc0055813ef"}
Oct 30 04:06:25.649873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.649845    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" event={"ID":"e1972c32-1497-43e9-9837-279d28a0b6ac","Type":"ContainerStarted","Data":"2e25c50995d2b98f59e16bba4aa927088bb410ffb91ef836b4daa89eda7d8ba6"}
Oct 30 04:06:25.650981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:25.650962    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerStarted","Data":"5600b0b8634f4e74e9b25c8386610223dc6d5cd5b6bd6333565cafe38dbfcbe2"}
Oct 30 04:06:26.894648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:26.894609    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:27.004616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.004558    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:27.769522 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.769471    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:27.868284 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.868255    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:06:27.871857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.871831    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.897705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.897634    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\""
Oct 30 04:06:27.897705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.897668    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\""
Oct 30 04:06:27.900879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.900803    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:06:27.930119 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4xp\" (UniqueName: \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.930315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930143    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.930315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930233    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.930438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.930438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930351    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.930438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.930385    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:27.942383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:27.942350    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031354    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031408    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4xp\" (UniqueName: \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031514    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.032109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.031543    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.033729 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.032722    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.033729 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.033312    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.035323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.035284    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.036495 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.036447    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.045976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.045951    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.055912 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.055875    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4xp\" (UniqueName: \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:28.114166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.114126    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:28.668886 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.668854    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerStarted","Data":"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"}
Oct 30 04:06:28.670220 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.670190    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" event={"ID":"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2","Type":"ContainerStarted","Data":"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"}
Oct 30 04:06:28.670344 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.670277    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:28.671570 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.671550    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" event={"ID":"e1972c32-1497-43e9-9837-279d28a0b6ac","Type":"ContainerStarted","Data":"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"}
Oct 30 04:06:28.671701 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.671684    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:28.712083 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.712035    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" podStartSLOduration=1.6318112280000001 podStartE2EDuration="4.712022328s" totalImagesPullingTime="3.0802111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:28.71133643 +0000 UTC m=+5146.935585707" watchObservedRunningTime="2026-10-30 04:06:28.712022328 +0000 UTC m=+5146.936271605"
Oct 30 04:06:28.752936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:28.752888    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" podStartSLOduration=2.009191583 podStartE2EDuration="4.752874746s" totalImagesPullingTime="2.743683163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:28.752096643 +0000 UTC m=+5146.976345920" watchObservedRunningTime="2026-10-30 04:06:28.752874746 +0000 UTC m=+5146.977124024"
Oct 30 04:06:30.683277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.683243    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerStarted","Data":"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"}
Oct 30 04:06:30.683723 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.683461    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:30.683723 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.683502    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:30.689587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.689566    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:30.692130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.692109    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:06:30.726492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:30.726385    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" podStartSLOduration=2.3488749540000002 podStartE2EDuration="6.726368536s" totalImagesPullingTime="4.377493582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:30.726071386 +0000 UTC m=+5148.950320697" watchObservedRunningTime="2026-10-30 04:06:30.726368536 +0000 UTC m=+5148.950617816"
Oct 30 04:06:32.263159 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:32.263111    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:32.290793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:32.290764    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:32.364284 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:32.364245    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:32.364473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:32.364379    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:32.378218 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:32.378184    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") DevicePath \"csi-9eab6a07b1d61d94cb8047377e7bdc52265c941fe4e8c4cb4476f1cdd006914d\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:33.545080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:33.545040    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d0ba80d454beb69db9470fc45e5a47dc3c954c5997bec8b3d02403b351bf919d/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:33.568748 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:33.568716    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:33.586779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:33.586751    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:33.890716 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:06:33.890687    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f127d0_a8a4_4aec_b07a_4061ba16d85e.slice/crio-2496477d6d29c1e602def326391455b306bda830b62968eeaaf2a2b8cfc4f3b7 WatchSource:0}: Error finding container 2496477d6d29c1e602def326391455b306bda830b62968eeaaf2a2b8cfc4f3b7: Status 404 returned error can't find the container with id 2496477d6d29c1e602def326391455b306bda830b62968eeaaf2a2b8cfc4f3b7
Oct 30 04:06:33.892898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:33.892874    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:06:34.631252 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:06:34.631215    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:06:34.701596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:34.701559    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"05f127d0-a8a4-4aec-b07a-4061ba16d85e","Type":"ContainerStarted","Data":"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"}
Oct 30 04:06:34.701795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:34.701644    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"05f127d0-a8a4-4aec-b07a-4061ba16d85e","Type":"ContainerStarted","Data":"2496477d6d29c1e602def326391455b306bda830b62968eeaaf2a2b8cfc4f3b7"}
Oct 30 04:06:34.701795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:34.701706    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:34.744880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:34.744811    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=10.744796813 podStartE2EDuration="10.744796813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:34.743971319 +0000 UTC m=+5152.968220597" watchObservedRunningTime="2026-10-30 04:06:34.744796813 +0000 UTC m=+5152.969046089"
Oct 30 04:06:40.239786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:40.239751    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:49.683791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:49.683757    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:06:49.684408 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:49.684382    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:06:55.711652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:55.711583    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:06:55.853094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:55.853054    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:55.979364 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:55.979273    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:58.153568 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:58.153510    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:58.273237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:58.273203    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:06:59.000344 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:06:59.000309    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:04.631965 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:07:04.631932    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:07:30.701211 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:30.701176    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:30.813138 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:30.813091    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:33.036813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:33.036772    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:33.150965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:33.150922    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:33.810124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:33.810093    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:33.931773 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:33.931739    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:34.630688 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:07:34.630658    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:07:40.383880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:40.383810    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:07:42.659749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:07:42.659717    4528 scope.go:122] "RemoveContainer" containerID="f760d209ce7c39adbac2b42de6213a74bec2704dde509c2b249c08b8673f8d77"
Oct 30 04:08:04.630685 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:08:04.630648    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:08:34.630190 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:08:34.630058    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:08:37.674048 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:37.674007    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:37.793989 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:37.793956    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:39.968562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:39.968527    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:40.105215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.105185    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:40.495920 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.495876    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:40.653145 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.653106    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:08:40.656305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.656283    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.682401 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.682377    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 04:08:40.682524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.682443    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 04:08:40.683834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.683801    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:08:40.779050 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.778973    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:08:40.871453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871422    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.871609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.871609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871557    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.871694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871621    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.871694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871646    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.871694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.871670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.972561 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972523    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972575    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972609    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972634    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972727    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.972758    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.973564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.973705 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.973631    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.975298 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.975270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.975394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.975340    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.975436 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.975425    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:40.995900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:40.995877    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.087735 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.087695    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.088181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.087839    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.127456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.127414    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.127456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.127421    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.188249 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.188216    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.188420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.188262    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.188420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.188349    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.188420 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.188374    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.202726 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.202696    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") DevicePath \"csi-279d65c9b0001c12f34d25871612c4352dc0b924e74c784b4b0acac8df8dde94\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.202842 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.202740    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") DevicePath \"csi-fb9b5754ab65534995f9f7404da2b60d38294418faeb91305eccc402e5246be2\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.440559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.440528    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5a18b684db142dd3a35cb91f8150351c9e560e77f88e25963e593bd05a144db/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:43.463512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:43.463487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:50.523203 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:50.523162    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/393980d41d60e4cbec658b78e9ee48927cfc5faa7d50a5969ab6365fd37edfe9/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:50.548652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:50.548618    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:50.566996 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:50.566962    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:50.880404 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:50.880306    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:08:51.275231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:51.275136    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"344db37a-ada2-4b49-80bd-a6c8cb0a24bb","Type":"ContainerStarted","Data":"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"}
Oct 30 04:08:51.275231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:51.275197    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"344db37a-ada2-4b49-80bd-a6c8cb0a24bb","Type":"ContainerStarted","Data":"ed9f9b05882646108813401ca0079841562b69383c7944d9617931bad1fb0e61"}
Oct 30 04:08:51.275417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:51.275235    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:08:51.319971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:08:51.319924    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podStartSLOduration=14.319910574 podStartE2EDuration="14.319910574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:08:51.318687446 +0000 UTC m=+5289.542936746" watchObservedRunningTime="2026-10-30 04:08:51.319910574 +0000 UTC m=+5289.544159852"
Oct 30 04:09:04.631107 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:09:04.631070    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:09:07.413449 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:07.413414    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:09.883445 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:09.883407    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:12.285493 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:12.285451    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:09:16.045450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:16.045415    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:16.167062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:16.167028    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:22.285660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:22.285603    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:09:32.283734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:32.283686    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:09:34.631111 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:09:34.631080    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:09:40.670543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:40.670507    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:42.284505 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:42.284476    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:09:51.831097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:51.831056    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:09:51.954119 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:09:51.954084    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:04.629950 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:04.629913    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:10:05.284756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:05.284721    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:05.401467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:05.401432    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:05.520543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:05.520515    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:10.203706 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.203670    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:10.204172 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.203880    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester" containerID="cri-o://d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109" gracePeriod=600
Oct 30 04:10:10.524175 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.524101    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:10.527488 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.527470    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.541656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.541626    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-94m9t\""
Oct 30 04:10:10.541786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.541716    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\""
Oct 30 04:10:10.549425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.549394    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:10.639142 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.639098    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.639381 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.639255    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wg5\" (UniqueName: \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.649654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.649623    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:10.740282 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.740245    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wg5\" (UniqueName: \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.740476 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.740314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.740903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.740884    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.763672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.763640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wg5\" (UniqueName: \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\") pod \"data-science-usage-logs-collector-54c8646c45-pp6qw\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:10.839859 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:10.839777    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:11.138233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.138209    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:11.140672 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:10:11.140642    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda83a369_d22c_4e4f_ba2f_b1474f9ac997.slice/crio-5ba88723fb0ddc60dbe8edc47f840db26176ea158c583ba1c87c8e360986e8c5 WatchSource:0}: Error finding container 5ba88723fb0ddc60dbe8edc47f840db26176ea158c583ba1c87c8e360986e8c5: Status 404 returned error can't find the container with id 5ba88723fb0ddc60dbe8edc47f840db26176ea158c583ba1c87c8e360986e8c5
Oct 30 04:10:11.143578 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.143539    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:10:11.143663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.143607    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:10:11.259451 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.259423    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.344303 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344260    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344303 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344299    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344549 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344549 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344435    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344549 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344500    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344549 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344540    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344585    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.344732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.344675    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\" (UID: \"344db37a-ada2-4b49-80bd-a6c8cb0a24bb\") "
Oct 30 04:10:11.345084 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.345055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:11.345565 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.345413    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:11.347747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.347721    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:11.347919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.347895    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:11.348167 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.348019    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "kube-api-access-bxbx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:11.348327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.348304    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:10:11.348468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.348447    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "pvc-5441af3d-1475-431a-b556-4acba3990893". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:10:11.348929 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.348905    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret" pod "344db37a-ada2-4b49-80bd-a6c8cb0a24bb" (UID: "344db37a-ada2-4b49-80bd-a6c8cb0a24bb"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:11.445972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.445936    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:10:11.445972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.445977    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.445997    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ingester-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.446013    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-test-loki-shared-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.446044    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.446059    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.446074    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.446232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.446089    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/344db37a-ada2-4b49-80bd-a6c8cb0a24bb-kube-api-access-bxbx8\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:11.455965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.455942    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5441af3d-1475-431a-b556-4acba3990893" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:10:11.459754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.459736    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:10:11.547130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.547099    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-279d65c9b0001c12f34d25871612c4352dc0b924e74c784b4b0acac8df8dde94\""
Oct 30 04:10:11.547130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.547126    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-fb9b5754ab65534995f9f7404da2b60d38294418faeb91305eccc402e5246be2\""
Oct 30 04:10:11.617407 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.617373    4528 generic.go:356] "Generic (PLEG): container finished" podID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerID="d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109" exitCode=0
Oct 30 04:10:11.617590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.617425    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.617590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.617451    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"344db37a-ada2-4b49-80bd-a6c8cb0a24bb","Type":"ContainerDied","Data":"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"}
Oct 30 04:10:11.617590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.617500    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"344db37a-ada2-4b49-80bd-a6c8cb0a24bb","Type":"ContainerDied","Data":"ed9f9b05882646108813401ca0079841562b69383c7944d9617931bad1fb0e61"}
Oct 30 04:10:11.617590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.617522    4528 scope.go:122] "RemoveContainer" containerID="d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"
Oct 30 04:10:11.619327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.619305    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" event={"ID":"da83a369-d22c-4e4f-ba2f-b1474f9ac997","Type":"ContainerStarted","Data":"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"}
Oct 30 04:10:11.619435 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.619347    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" event={"ID":"da83a369-d22c-4e4f-ba2f-b1474f9ac997","Type":"ContainerStarted","Data":"5ba88723fb0ddc60dbe8edc47f840db26176ea158c583ba1c87c8e360986e8c5"}
Oct 30 04:10:11.627488 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.627472    4528 scope.go:122] "RemoveContainer" containerID="d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"
Oct 30 04:10:11.627749 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:11.627731    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109\": container with ID starting with d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109 not found: ID does not exist" containerID="d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"
Oct 30 04:10:11.627787 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.627756    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109"} err="failed to get container status \"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109\": rpc error: code = NotFound desc = could not find container \"d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109\": container with ID starting with d0d42594207756a2a4597c3052c31f1fdd40931721ad8ae0452562698dfe0109 not found: ID does not exist"
Oct 30 04:10:11.671107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.670881    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" podStartSLOduration=1.67086371 podStartE2EDuration="1.67086371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:11.670253524 +0000 UTC m=+5369.894502801" watchObservedRunningTime="2026-10-30 04:10:11.67086371 +0000 UTC m=+5369.895112988"
Oct 30 04:10:11.686493 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.686464    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:10:11.686805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.686788    4528 cpu_manager.go:414] "containerMap: removing container" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester"
Oct 30 04:10:11.686805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.686805    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester"
Oct 30 04:10:11.687008 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.686908    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" containerName="loki-ingester"
Oct 30 04:10:11.691658 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.691635    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.708933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.708901    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:11.714299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.714261    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\""
Oct 30 04:10:11.714299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.714261    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\""
Oct 30 04:10:11.714496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.714481    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\""
Oct 30 04:10:11.716085 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.716061    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:10:11.719153 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.719127    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:11.748994 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.748960    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.749195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.749002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.749195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.749031    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.749195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.749068    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.749195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.749118    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4ph6\" (UniqueName: \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.749393 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.749298    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.765621 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.765588    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:11.775472 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.774796    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.791753 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.791725    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:11.812866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.812836    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 04:10:11.812866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.812847    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 04:10:11.850724 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.850908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850732    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4ph6\" (UniqueName: \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.850908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850762    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.850908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850882    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.851107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850925    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.851107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.850949    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.851689 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.851656    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.851838 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.851743    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.853347 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.853323    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.853485 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.853463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.853611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.853591    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.873248 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.873222    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4ph6\" (UniqueName: \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:11.893511 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.893480    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344db37a-ada2-4b49-80bd-a6c8cb0a24bb" path="/var/lib/kubelet/pods/344db37a-ada2-4b49-80bd-a6c8cb0a24bb/volumes"
Oct 30 04:10:11.951849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.951754    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.951849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.951794    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.951849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.951830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.952061 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.951896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.952061 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.952013    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzq5x\" (UniqueName: \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:11.952061 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:11.952036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053103 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053103 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053108    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053129    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053241    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053327    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzq5x\" (UniqueName: \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.053502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.053361    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.054031 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.054002    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.054173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.054055    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.055782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.055757    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.056151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.056131    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.056267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.056246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.076214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.076192    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzq5x\" (UniqueName: \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:12.406346 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:12.406309    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:13.392223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.392183    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:13.442467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.442433    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:13.571950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.571918    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:13.628661 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.628619    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" podUID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" containerName="otc-container" containerID="cri-o://dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27" gracePeriod=30
Oct 30 04:10:13.889856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.889803    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:13.971028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.970953    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\" (UniqueName: \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\") pod \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") "
Oct 30 04:10:13.971028 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.971018    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\" (UniqueName: \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\") pod \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\" (UID: \"da83a369-d22c-4e4f-ba2f-b1474f9ac997\") "
Oct 30 04:10:13.971474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.971401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal" pod "da83a369-d22c-4e4f-ba2f-b1474f9ac997" (UID: "da83a369-d22c-4e4f-ba2f-b1474f9ac997"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:13.973189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:13.973169    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5" pod "da83a369-d22c-4e4f-ba2f-b1474f9ac997" (UID: "da83a369-d22c-4e4f-ba2f-b1474f9ac997"). InnerVolumeSpecName "kube-api-access-w5wg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:14.071988 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.071946    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/da83a369-d22c-4e4f-ba2f-b1474f9ac997-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:14.071988 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.071977    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5wg5\" (UniqueName: \"kubernetes.io/projected/da83a369-d22c-4e4f-ba2f-b1474f9ac997-kube-api-access-w5wg5\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:14.476606 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.476563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.477054 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.476634    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.504573 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.504541    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.504573 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.504549    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.578108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.578069    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.578322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.578118    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.578322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.578219    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.578322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.578228    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.592219 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.592193    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") DevicePath \"csi-fb9b5754ab65534995f9f7404da2b60d38294418faeb91305eccc402e5246be2\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.592302 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.592191    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") DevicePath \"csi-279d65c9b0001c12f34d25871612c4352dc0b924e74c784b4b0acac8df8dde94\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.634078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.634049    4528 generic.go:356] "Generic (PLEG): container finished" podID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" containerID="dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27" exitCode=0
Oct 30 04:10:14.634244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.634088    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"
Oct 30 04:10:14.634244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.634121    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" event={"ID":"da83a369-d22c-4e4f-ba2f-b1474f9ac997","Type":"ContainerDied","Data":"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"}
Oct 30 04:10:14.634244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.634175    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw" event={"ID":"da83a369-d22c-4e4f-ba2f-b1474f9ac997","Type":"ContainerDied","Data":"5ba88723fb0ddc60dbe8edc47f840db26176ea158c583ba1c87c8e360986e8c5"}
Oct 30 04:10:14.634244 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.634200    4528 scope.go:122] "RemoveContainer" containerID="dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"
Oct 30 04:10:14.643378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.643362    4528 scope.go:122] "RemoveContainer" containerID="dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"
Oct 30 04:10:14.643648 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:14.643628    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27\": container with ID starting with dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27 not found: ID does not exist" containerID="dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"
Oct 30 04:10:14.643699 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.643655    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27"} err="failed to get container status \"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27\": rpc error: code = NotFound desc = could not find container \"dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27\": container with ID starting with dd852f1c47cc1d261a44ce869d6a23c8d8729608c014afb1cf2c6304683dec27 not found: ID does not exist"
Oct 30 04:10:14.682491 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.682466    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:14.686937 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.686911    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-pp6qw"]
Oct 30 04:10:14.690621 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.690597    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b5a18b684db142dd3a35cb91f8150351c9e560e77f88e25963e593bd05a144db/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.704030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.704002    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/393980d41d60e4cbec658b78e9ee48927cfc5faa7d50a5969ab6365fd37edfe9/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.719466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.719445    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.726829 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.726769    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"data-science-lokistack-ingester-2\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:14.792951 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:14.792916    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:15.106922 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.106894    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:10:15.109186 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:10:15.109156    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee80cd6_0004_4966_8ad0_2f6e9fecc51d.slice/crio-ed0fc94919e88186b241ab76588be270563d76579fc404fd45b43ac6abbd3c66 WatchSource:0}: Error finding container ed0fc94919e88186b241ab76588be270563d76579fc404fd45b43ac6abbd3c66: Status 404 returned error can't find the container with id ed0fc94919e88186b241ab76588be270563d76579fc404fd45b43ac6abbd3c66
Oct 30 04:10:15.640778 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.640736    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"bee80cd6-0004-4966-8ad0-2f6e9fecc51d","Type":"ContainerStarted","Data":"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"}
Oct 30 04:10:15.641229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.640836    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"bee80cd6-0004-4966-8ad0-2f6e9fecc51d","Type":"ContainerStarted","Data":"ed0fc94919e88186b241ab76588be270563d76579fc404fd45b43ac6abbd3c66"}
Oct 30 04:10:15.641229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.640869    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:15.682936 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.682879    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podStartSLOduration=4.682866254 podStartE2EDuration="4.682866254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:15.68187082 +0000 UTC m=+5373.906120095" watchObservedRunningTime="2026-10-30 04:10:15.682866254 +0000 UTC m=+5373.907115521"
Oct 30 04:10:15.891969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:15.891895    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" path="/var/lib/kubelet/pods/da83a369-d22c-4e4f-ba2f-b1474f9ac997/volumes"
Oct 30 04:10:28.799464 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:28.799429    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:28.827129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:28.827098    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:28.899903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:28.899872    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:28.900057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:28.899999    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:28.913863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:28.913839    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") DevicePath \"csi-ebd4886a9a4100739c9334fb8333d3168f81af4cf4144a232f334726114de08a\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:29.015422 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.015393    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2a84ccb0e632fc97a562bedb1c2138c4534d44455a54f3f6cb9692580fc3a7df/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:29.043871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.043835    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"data-science-lokistack-compactor-0\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:29.104063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.103985    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:29.405296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.405254    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:10:29.408400 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:10:29.408371    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a66854b_8efa_43d6_aa8d_8b3f2b499e57.slice/crio-87127848379aa37461961117b9822759beb0ae343a6e2d5e2bec7402643ccb97 WatchSource:0}: Error finding container 87127848379aa37461961117b9822759beb0ae343a6e2d5e2bec7402643ccb97: Status 404 returned error can't find the container with id 87127848379aa37461961117b9822759beb0ae343a6e2d5e2bec7402643ccb97
Oct 30 04:10:29.701508 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.701431    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"4a66854b-8efa-43d6-aa8d-8b3f2b499e57","Type":"ContainerStarted","Data":"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"}
Oct 30 04:10:29.701508 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.701480    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"4a66854b-8efa-43d6-aa8d-8b3f2b499e57","Type":"ContainerStarted","Data":"87127848379aa37461961117b9822759beb0ae343a6e2d5e2bec7402643ccb97"}
Oct 30 04:10:29.701696 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.701565    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:29.745301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:29.745250    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=18.745235292 podStartE2EDuration="18.745235292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:29.743056698 +0000 UTC m=+5387.967305998" watchObservedRunningTime="2026-10-30 04:10:29.745235292 +0000 UTC m=+5387.969484572"
Oct 30 04:10:34.631685 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:34.631650    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:10:36.649882 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:36.649846    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:40.761351 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:40.761309    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:10:42.690730 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:42.690697    4528 scope.go:122] "RemoveContainer" containerID="a7dd66f91f71656cfca6f97735889ad6c46b86a70a05fec33102c89b228b51ba"
Oct 30 04:10:42.699404 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:42.699386    4528 scope.go:122] "RemoveContainer" containerID="38989f577be8d4a7c3a7136042d14cdefdc4773277195c4936c133885dabde4b"
Oct 30 04:10:42.706973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:42.706952    4528 scope.go:122] "RemoveContainer" containerID="cd6ec47cc11538cf5376518fd060ac10c3f2583218eee29f143d3d282fc8e53c"
Oct 30 04:10:46.314015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.313984    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:10:46.314484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.314191    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" containerName="loki-index-gateway" containerID="cri-o://e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b" gracePeriod=30
Oct 30 04:10:46.577388 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.577333    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:46.650295 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.650251    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:46.660971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.660943    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661015    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\" (UniqueName: \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661039    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661154    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661215    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661241    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\") pod \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\" (UID: \"05f127d0-a8a4-4aec-b07a-4061ba16d85e\") "
Oct 30 04:10:46.661952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661398    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:46.661952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661496    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.661952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.661945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:46.663759 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.663725    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:46.663966 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.663774    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "kube-api-access-8s4xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:46.664190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.664023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:46.664190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.664068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:46.664372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.664347    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a" pod "05f127d0-a8a4-4aec-b07a-4061ba16d85e" (UID: "05f127d0-a8a4-4aec-b07a-4061ba16d85e"). InnerVolumeSpecName "pvc-536701e3-6ae5-4168-ad78-e30a293330df". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:10:46.761905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761871    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4xp\" (UniqueName: \"kubernetes.io/projected/05f127d0-a8a4-4aec-b07a-4061ba16d85e-kube-api-access-8s4xp\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.761905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761899    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.762114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761930    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:10:46.762114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761939    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-test-loki-shared-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.762114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761950    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.762114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.761960    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/05f127d0-a8a4-4aec-b07a-4061ba16d85e-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:46.770739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.770718    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-536701e3-6ae5-4168-ad78-e30a293330df" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:10:46.775528 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.775503    4528 generic.go:356] "Generic (PLEG): container finished" podID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" containerID="e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b" exitCode=0
Oct 30 04:10:46.775645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.775536    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"05f127d0-a8a4-4aec-b07a-4061ba16d85e","Type":"ContainerDied","Data":"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"}
Oct 30 04:10:46.775645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.775552    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:10:46.775645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.775574    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"05f127d0-a8a4-4aec-b07a-4061ba16d85e","Type":"ContainerDied","Data":"2496477d6d29c1e602def326391455b306bda830b62968eeaaf2a2b8cfc4f3b7"}
Oct 30 04:10:46.775645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.775593    4528 scope.go:122] "RemoveContainer" containerID="e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"
Oct 30 04:10:46.785120 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.785101    4528 scope.go:122] "RemoveContainer" containerID="e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"
Oct 30 04:10:46.785406 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:46.785387    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b\": container with ID starting with e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b not found: ID does not exist" containerID="e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"
Oct 30 04:10:46.785456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.785413    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b"} err="failed to get container status \"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b\": rpc error: code = NotFound desc = could not find container \"e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b\": container with ID starting with e556eb2645231c5e35197d81f2d410e86e238e5b08999a989b4594f25095a88b not found: ID does not exist"
Oct 30 04:10:46.827481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.827413    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:10:46.834116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.834093    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:10:46.863009 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:46.862977    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-9eab6a07b1d61d94cb8047377e7bdc52265c941fe4e8c4cb4476f1cdd006914d\""
Oct 30 04:10:47.892568 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:47.892531    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" path="/var/lib/kubelet/pods/05f127d0-a8a4-4aec-b07a-4061ba16d85e/volumes"
Oct 30 04:10:50.710742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:50.710716    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:10:55.378919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:55.378881    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:10:55.379450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:55.379089    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" podUID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" containerName="loki-querier" containerID="cri-o://3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff" gracePeriod=30
Oct 30 04:10:56.156427 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.156401    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:10:56.245277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245201    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245242    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245314    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\" (UniqueName: \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245337    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245394    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245431    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\") pod \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\" (UID: \"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2\") "
Oct 30 04:10:56.245921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245890    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:56.246047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.245884    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:56.247496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.247469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:56.247584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.247523    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "kube-api-access-czhrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:56.247584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.247544    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "test-loki-shared-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:56.247664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.247621    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc" pod "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" (UID: "e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:56.346238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346201    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhrl\" (UniqueName: \"kubernetes.io/projected/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-kube-api-access-czhrl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.346238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346231    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.346238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346241    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.346469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346255    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.346469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346264    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-data-science-lokistack-querier-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.346469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.346273    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-shared-secret\" (UniqueName: \"kubernetes.io/secret/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2-test-loki-shared-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:10:56.649351 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.649318    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:10:56.822477 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.822441    4528 generic.go:356] "Generic (PLEG): container finished" podID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" containerID="3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff" exitCode=0
Oct 30 04:10:56.822663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.822489    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" event={"ID":"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2","Type":"ContainerDied","Data":"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"}
Oct 30 04:10:56.822663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.822509    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"
Oct 30 04:10:56.822663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.822552    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz" event={"ID":"e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2","Type":"ContainerDied","Data":"9100aef4fe4c574837f7b664028d71399367e7a58f99c877f5628dc0055813ef"}
Oct 30 04:10:56.822663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.822572    4528 scope.go:122] "RemoveContainer" containerID="3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"
Oct 30 04:10:56.831011 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.830993    4528 scope.go:122] "RemoveContainer" containerID="3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"
Oct 30 04:10:56.831467 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:10:56.831275    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff\": container with ID starting with 3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff not found: ID does not exist" containerID="3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"
Oct 30 04:10:56.831467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.831301    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff"} err="failed to get container status \"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff\": rpc error: code = NotFound desc = could not find container \"3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff\": container with ID starting with 3741b7f09490d2b73d4799b8dba25787d0484c030e3a07d42848fd7050abbfff not found: ID does not exist"
Oct 30 04:10:56.871541 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.871506    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:10:56.877536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:56.877514    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-84985db556-tx4kz"]
Oct 30 04:10:57.892248 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:10:57.892213    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" path="/var/lib/kubelet/pods/e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2/volumes"
Oct 30 04:11:04.631626 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:11:04.631591    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:11:34.630996 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:11:34.630953    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:11:40.929733 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:11:40.929693    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:11:45.042293 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:11:45.042234    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:04.630986 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:04.630951    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:12:34.631331 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:34.631288    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:12:41.052796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:41.052756    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:53.622071 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622034    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:53.622611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" containerName="otc-container"
Oct 30 04:12:53.622659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622616    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" containerName="otc-container"
Oct 30 04:12:53.622659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622633    4528 cpu_manager.go:414] "containerMap: removing container" podUID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" containerName="loki-index-gateway"
Oct 30 04:12:53.622659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622643    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" containerName="loki-index-gateway"
Oct 30 04:12:53.622768 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622679    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" containerName="loki-querier"
Oct 30 04:12:53.622768 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622688    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" containerName="loki-querier"
Oct 30 04:12:53.622849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622779    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="da83a369-d22c-4e4f-ba2f-b1474f9ac997" containerName="otc-container"
Oct 30 04:12:53.622849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622791    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f127d0-a8a4-4aec-b07a-4061ba16d85e" containerName="loki-index-gateway"
Oct 30 04:12:53.622849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.622803    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ed00fd-b3ff-46ce-a12d-c1c529f72cc2" containerName="loki-querier"
Oct 30 04:12:53.626159 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.626139    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.641463 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.641440    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.641606 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.641469    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8sl\" (UniqueName: \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.645376 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.645347    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:53.653552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.653144    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rxgsr\""
Oct 30 04:12:53.653693 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.653367    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\""
Oct 30 04:12:53.666071 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.666046    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:53.742221 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.742140    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.742221 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.742177    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8sl\" (UniqueName: \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.742796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.742775    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.766650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.766619    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8sl\" (UniqueName: \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\") pod \"data-science-usage-logs-collector-54c8646c45-xvvsh\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:53.799392 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.799358    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:53.939857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:53.939798    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:54.235873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.235849    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:54.238537 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:12:54.238509    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e3de77_a015_46f7_b970_a14b9d6451cf.slice/crio-890791c740a0b683f3ed9e34b46fc4823baf5a68711a2860648bddc245737b9f WatchSource:0}: Error finding container 890791c740a0b683f3ed9e34b46fc4823baf5a68711a2860648bddc245737b9f: Status 404 returned error can't find the container with id 890791c740a0b683f3ed9e34b46fc4823baf5a68711a2860648bddc245737b9f
Oct 30 04:12:54.240727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.240695    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:12:54.240867 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.240750    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:12:54.309097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.309063    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" event={"ID":"76e3de77-a015-46f7-b970-a14b9d6451cf","Type":"ContainerStarted","Data":"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"}
Oct 30 04:12:54.309246 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.309120    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" event={"ID":"76e3de77-a015-46f7-b970-a14b9d6451cf","Type":"ContainerStarted","Data":"890791c740a0b683f3ed9e34b46fc4823baf5a68711a2860648bddc245737b9f"}
Oct 30 04:12:54.349528 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:54.349473    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" podStartSLOduration=1.349459907 podStartE2EDuration="1.349459907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:54.349319342 +0000 UTC m=+5532.573568620" watchObservedRunningTime="2026-10-30 04:12:54.349459907 +0000 UTC m=+5532.573709181"
Oct 30 04:12:57.514516 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.514479    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:57.579047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.579010    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:57.579568 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.579356    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester" containerID="cri-o://c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0" gracePeriod=600
Oct 30 04:12:57.615784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.615745    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:57.616015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.615987    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" containerName="loki-compactor" containerID="cri-o://ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3" gracePeriod=30
Oct 30 04:12:57.628379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.628349    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:12:57.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.628653    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="opa" containerID="cri-o://97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14" gracePeriod=30
Oct 30 04:12:57.629509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.629447    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="gateway" containerID="cri-o://0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b" gracePeriod=30
Oct 30 04:12:57.633625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.633598    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:12:57.633857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.633801    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" podUID="e1972c32-1497-43e9-9837-279d28a0b6ac" containerName="loki-distributor" containerID="cri-o://a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01" gracePeriod=30
Oct 30 04:12:57.653547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.653505    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:57.778269 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.778185    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:57.778433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.778370    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" podUID="76e3de77-a015-46f7-b970-a14b9d6451cf" containerName="otc-container" containerID="cri-o://7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb" gracePeriod=30
Oct 30 04:12:57.822004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.821966    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:57.948131 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:57.948096    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:12:58.062422 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.062399    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:58.183089 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.183056    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\") pod \"76e3de77-a015-46f7-b970-a14b9d6451cf\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") "
Oct 30 04:12:58.183261 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.183210    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\" (UniqueName: \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\") pod \"76e3de77-a015-46f7-b970-a14b9d6451cf\" (UID: \"76e3de77-a015-46f7-b970-a14b9d6451cf\") "
Oct 30 04:12:58.183493 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.183469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal" pod "76e3de77-a015-46f7-b970-a14b9d6451cf" (UID: "76e3de77-a015-46f7-b970-a14b9d6451cf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.185281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.185256    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl" pod "76e3de77-a015-46f7-b970-a14b9d6451cf" (UID: "76e3de77-a015-46f7-b970-a14b9d6451cf"). InnerVolumeSpecName "kube-api-access-8l8sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.284995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.284950    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8sl\" (UniqueName: \"kubernetes.io/projected/76e3de77-a015-46f7-b970-a14b9d6451cf-kube-api-access-8l8sl\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.284995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.284986    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/76e3de77-a015-46f7-b970-a14b9d6451cf-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.329119 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.329088    4528 generic.go:356] "Generic (PLEG): container finished" podID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerID="97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14" exitCode=0
Oct 30 04:12:58.329276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.329162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerDied","Data":"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"}
Oct 30 04:12:58.330480 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.330458    4528 generic.go:356] "Generic (PLEG): container finished" podID="76e3de77-a015-46f7-b970-a14b9d6451cf" containerID="7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb" exitCode=0
Oct 30 04:12:58.330623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.330493    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" event={"ID":"76e3de77-a015-46f7-b970-a14b9d6451cf","Type":"ContainerDied","Data":"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"}
Oct 30 04:12:58.330623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.330512    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"
Oct 30 04:12:58.330623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.330538    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh" event={"ID":"76e3de77-a015-46f7-b970-a14b9d6451cf","Type":"ContainerDied","Data":"890791c740a0b683f3ed9e34b46fc4823baf5a68711a2860648bddc245737b9f"}
Oct 30 04:12:58.330623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.330563    4528 scope.go:122] "RemoveContainer" containerID="7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"
Oct 30 04:12:58.387290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.387264    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:58.392367 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.392345    4528 scope.go:122] "RemoveContainer" containerID="7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"
Oct 30 04:12:58.392796 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:58.392767    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb\": container with ID starting with 7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb not found: ID does not exist" containerID="7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"
Oct 30 04:12:58.392918 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.392810    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb"} err="failed to get container status \"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb\": rpc error: code = NotFound desc = could not find container \"7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb\": container with ID starting with 7def2f888cd17701c82d3dfe3f8b7b66a487bf60d5ddea250eca3aed4e9ed8eb not found: ID does not exist"
Oct 30 04:12:58.398580 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.398560    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-54c8646c45-xvvsh"]
Oct 30 04:12:58.546236 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.546211    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:12:58.587963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.587924    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\" (UniqueName: \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.588126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.587996    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.588126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588107    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.588255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588138    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.589177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588341    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.589177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.589177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588424    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.589177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.588462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\") pod \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\" (UID: \"bee80cd6-0004-4966-8ad0-2f6e9fecc51d\") "
Oct 30 04:12:58.589485 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.589264    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.589541 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.589481    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.599997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.593434    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.605072 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.604832    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.606302 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.606255    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "kube-api-access-rzq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.606616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.606594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:58.606718 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.606686    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.607033 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.607010    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050" pod "bee80cd6-0004-4966-8ad0-2f6e9fecc51d" (UID: "bee80cd6-0004-4966-8ad0-2f6e9fecc51d"). InnerVolumeSpecName "pvc-5441af3d-1475-431a-b556-4acba3990893". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:58.618073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.618049    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:12:58.620156 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.620136    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:12:58.690333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\") pod \"e1972c32-1497-43e9-9837-279d28a0b6ac\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") "
Oct 30 04:12:58.690500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690340    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\") pod \"e1972c32-1497-43e9-9837-279d28a0b6ac\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") "
Oct 30 04:12:58.690500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690361    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\") pod \"e1972c32-1497-43e9-9837-279d28a0b6ac\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") "
Oct 30 04:12:58.690500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690386    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.690669 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690531    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\") pod \"e1972c32-1497-43e9-9837-279d28a0b6ac\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") "
Oct 30 04:12:58.690669 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\" (UniqueName: \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\") pod \"e1972c32-1497-43e9-9837-279d28a0b6ac\" (UID: \"e1972c32-1497-43e9-9837-279d28a0b6ac\") "
Oct 30 04:12:58.690784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690672    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.690784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690704    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.690784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690746    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config" pod "e1972c32-1497-43e9-9837-279d28a0b6ac" (UID: "e1972c32-1497-43e9-9837-279d28a0b6ac"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.691003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690781    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle" pod "e1972c32-1497-43e9-9837-279d28a0b6ac" (UID: "e1972c32-1497-43e9-9837-279d28a0b6ac"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.691003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690813    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.691003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690922    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\" (UniqueName: \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.691003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690951    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.691003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.690979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\") pod \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\" (UID: \"4a66854b-8efa-43d6-aa8d-8b3f2b499e57\") "
Oct 30 04:12:58.691278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691239    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.691278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691256    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-test-loki-lifecycle-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691288    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691306    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691331    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691347    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691362    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ingester-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691377    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691391    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691405    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.691807 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691419    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzq5x\" (UniqueName: \"kubernetes.io/projected/bee80cd6-0004-4966-8ad0-2f6e9fecc51d-kube-api-access-rzq5x\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.692008 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.691902    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.694433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694352    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http" pod "e1972c32-1497-43e9-9837-279d28a0b6ac" (UID: "e1972c32-1497-43e9-9837-279d28a0b6ac"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.694433 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694378    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc" pod "e1972c32-1497-43e9-9837-279d28a0b6ac" (UID: "e1972c32-1497-43e9-9837-279d28a0b6ac"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.694713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694628    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.694713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694635    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "kube-api-access-g4ph6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.694993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694970    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.695082 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.694978    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:12:58.695166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.695149    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m" pod "e1972c32-1497-43e9-9837-279d28a0b6ac" (UID: "e1972c32-1497-43e9-9837-279d28a0b6ac"). InnerVolumeSpecName "kube-api-access-msg7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.695619 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.695594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc" pod "4a66854b-8efa-43d6-aa8d-8b3f2b499e57" (UID: "4a66854b-8efa-43d6-aa8d-8b3f2b499e57"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.702406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.702389    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:12:58.702789 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.702775    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-5441af3d-1475-431a-b556-4acba3990893" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:12:58.792933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792897    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4ph6\" (UniqueName: \"kubernetes.io/projected/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-kube-api-access-g4ph6\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.792933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792934    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-test-loki-lifecycle-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792949    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792966    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-44ff9f9a-2455-4861-9e52-f12dd3937aa3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0da349e22599e4d61\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-fb9b5754ab65534995f9f7404da2b60d38294418faeb91305eccc402e5246be2\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792982    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.792997    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793011    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/e1972c32-1497-43e9-9837-279d28a0b6ac-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793026    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msg7m\" (UniqueName: \"kubernetes.io/projected/e1972c32-1497-43e9-9837-279d28a0b6ac-kube-api-access-msg7m\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793039    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-compactor-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793054    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a66854b-8efa-43d6-aa8d-8b3f2b499e57-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793069    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-5441af3d-1475-431a-b556-4acba3990893\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-083a7dbead8cc3050\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-279d65c9b0001c12f34d25871612c4352dc0b924e74c784b4b0acac8df8dde94\""
Oct 30 04:12:58.793129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.793111    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:12:58.803906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.803853    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:12:58.870281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.870256    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:12:58.894399 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.894375    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-f6f9e4e0-1512-40d3-9ab7-a48e3d6a6657\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08842b709e4a30a1e\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-ebd4886a9a4100739c9334fb8333d3168f81af4cf4144a232f334726114de08a\""
Oct 30 04:12:58.995070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995036    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995127    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995151    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995202    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995225    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995314    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\") pod \"c6ff0038-854c-4145-948e-5f794ba65fcd\" (UID: \"c6ff0038-854c-4145-948e-5f794ba65fcd\") "
Oct 30 04:12:58.995916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.995885    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.997810 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.997776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.997810 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.997792    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:58.997985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.997903    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.997985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.997956    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "kube-api-access-rjzgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:58.999795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.999770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:58.999899 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:58.999866    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:59.001279 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.001258    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants" pod "c6ff0038-854c-4145-948e-5f794ba65fcd" (UID: "c6ff0038-854c-4145-948e-5f794ba65fcd"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096565    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096598    4528 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096608    4528 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-lokistack-gateway\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096620    4528 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096630    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096639    4528 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-tls-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096647    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/c6ff0038-854c-4145-948e-5f794ba65fcd-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.096651 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.096656    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjzgr\" (UniqueName: \"kubernetes.io/projected/c6ff0038-854c-4145-948e-5f794ba65fcd-kube-api-access-rjzgr\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:12:59.335526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.335490    4528 generic.go:356] "Generic (PLEG): container finished" podID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" containerID="ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3" exitCode=0
Oct 30 04:12:59.335713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.335537    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0"
Oct 30 04:12:59.335713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.335558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"4a66854b-8efa-43d6-aa8d-8b3f2b499e57","Type":"ContainerDied","Data":"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"}
Oct 30 04:12:59.335713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.335623    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"4a66854b-8efa-43d6-aa8d-8b3f2b499e57","Type":"ContainerDied","Data":"87127848379aa37461961117b9822759beb0ae343a6e2d5e2bec7402643ccb97"}
Oct 30 04:12:59.335713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.335650    4528 scope.go:122] "RemoveContainer" containerID="ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"
Oct 30 04:12:59.336977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.336951    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1972c32-1497-43e9-9837-279d28a0b6ac" containerID="a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01" exitCode=0
Oct 30 04:12:59.337113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.337009    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"
Oct 30 04:12:59.337113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.337049    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" event={"ID":"e1972c32-1497-43e9-9837-279d28a0b6ac","Type":"ContainerDied","Data":"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"}
Oct 30 04:12:59.337113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.337089    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m" event={"ID":"e1972c32-1497-43e9-9837-279d28a0b6ac","Type":"ContainerDied","Data":"2e25c50995d2b98f59e16bba4aa927088bb410ffb91ef836b4daa89eda7d8ba6"}
Oct 30 04:12:59.345457 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.345265    4528 scope.go:122] "RemoveContainer" containerID="ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"
Oct 30 04:12:59.345590 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:59.345560    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3\": container with ID starting with ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3 not found: ID does not exist" containerID="ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"
Oct 30 04:12:59.345703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.345599    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3"} err="failed to get container status \"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3\": rpc error: code = NotFound desc = could not find container \"ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3\": container with ID starting with ca31897c27b9f819aceba4a3978cfe2556db12f6effaeb56f2afceb2a8d461f3 not found: ID does not exist"
Oct 30 04:12:59.345703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.345621    4528 scope.go:122] "RemoveContainer" containerID="a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"
Oct 30 04:12:59.346021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.346000    4528 generic.go:356] "Generic (PLEG): container finished" podID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerID="c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0" exitCode=0
Oct 30 04:12:59.346111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.346054    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2"
Oct 30 04:12:59.346111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.346100    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"bee80cd6-0004-4966-8ad0-2f6e9fecc51d","Type":"ContainerDied","Data":"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"}
Oct 30 04:12:59.346223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.346144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-2" event={"ID":"bee80cd6-0004-4966-8ad0-2f6e9fecc51d","Type":"ContainerDied","Data":"ed0fc94919e88186b241ab76588be270563d76579fc404fd45b43ac6abbd3c66"}
Oct 30 04:12:59.348495 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.348432    4528 generic.go:356] "Generic (PLEG): container finished" podID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerID="0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b" exitCode=0
Oct 30 04:12:59.348495 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.348484    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"
Oct 30 04:12:59.348657 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.348508    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerDied","Data":"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"}
Oct 30 04:12:59.348657 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.348542    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28" event={"ID":"c6ff0038-854c-4145-948e-5f794ba65fcd","Type":"ContainerDied","Data":"5600b0b8634f4e74e9b25c8386610223dc6d5cd5b6bd6333565cafe38dbfcbe2"}
Oct 30 04:12:59.355032 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.355013    4528 scope.go:122] "RemoveContainer" containerID="a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"
Oct 30 04:12:59.355319 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:59.355296    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01\": container with ID starting with a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01 not found: ID does not exist" containerID="a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"
Oct 30 04:12:59.355372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.355329    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01"} err="failed to get container status \"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01\": rpc error: code = NotFound desc = could not find container \"a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01\": container with ID starting with a356827867a9cc81ef435552b6c007c69e07978d2d43284f9a337fa9248e1c01 not found: ID does not exist"
Oct 30 04:12:59.355372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.355354    4528 scope.go:122] "RemoveContainer" containerID="c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"
Oct 30 04:12:59.364057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.364037    4528 scope.go:122] "RemoveContainer" containerID="c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"
Oct 30 04:12:59.364352 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:59.364335    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0\": container with ID starting with c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0 not found: ID does not exist" containerID="c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"
Oct 30 04:12:59.364407 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.364361    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0"} err="failed to get container status \"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0\": rpc error: code = NotFound desc = could not find container \"c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0\": container with ID starting with c4330f77676d1185b3f0994a629dae0677307dd948e649d2a32edcd32fdf22a0 not found: ID does not exist"
Oct 30 04:12:59.364407 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.364383    4528 scope.go:122] "RemoveContainer" containerID="97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"
Oct 30 04:12:59.371608 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.371588    4528 scope.go:122] "RemoveContainer" containerID="0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"
Oct 30 04:12:59.379030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.379007    4528 scope.go:122] "RemoveContainer" containerID="97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"
Oct 30 04:12:59.379300 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:59.379280    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14\": container with ID starting with 97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14 not found: ID does not exist" containerID="97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"
Oct 30 04:12:59.379378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.379305    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14"} err="failed to get container status \"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14\": rpc error: code = NotFound desc = could not find container \"97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14\": container with ID starting with 97a93e62466a53da9a3008d91ee71367f240b1dce1697641389e371f277f2d14 not found: ID does not exist"
Oct 30 04:12:59.379378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.379321    4528 scope.go:122] "RemoveContainer" containerID="0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"
Oct 30 04:12:59.379596 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:12:59.379578    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b\": container with ID starting with 0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b not found: ID does not exist" containerID="0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"
Oct 30 04:12:59.379635 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.379606    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b"} err="failed to get container status \"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b\": rpc error: code = NotFound desc = could not find container \"0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b\": container with ID starting with 0c8f3213a59e1bd3464f757dd93139957f5fc875cfabc90315a5b72db335d01b not found: ID does not exist"
Oct 30 04:12:59.413660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.413623    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:59.418114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.418087    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"]
Oct 30 04:12:59.465354 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.465323    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:12:59.469265 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.469242    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-d7f6645-c4n28"]
Oct 30 04:12:59.507720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.507683    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:12:59.511809 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.511785    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-v9w9m"]
Oct 30 04:12:59.544760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.544734    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:59.548879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.548857    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-2"]
Oct 30 04:12:59.892359 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.892284    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" path="/var/lib/kubelet/pods/4a66854b-8efa-43d6-aa8d-8b3f2b499e57/volumes"
Oct 30 04:12:59.892642 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.892628    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e3de77-a015-46f7-b970-a14b9d6451cf" path="/var/lib/kubelet/pods/76e3de77-a015-46f7-b970-a14b9d6451cf/volumes"
Oct 30 04:12:59.893013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.893000    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" path="/var/lib/kubelet/pods/bee80cd6-0004-4966-8ad0-2f6e9fecc51d/volumes"
Oct 30 04:12:59.893665 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.893650    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" path="/var/lib/kubelet/pods/c6ff0038-854c-4145-948e-5f794ba65fcd/volumes"
Oct 30 04:12:59.894058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:12:59.894045    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1972c32-1497-43e9-9837-279d28a0b6ac" path="/var/lib/kubelet/pods/e1972c32-1497-43e9-9837-279d28a0b6ac/volumes"
Oct 30 04:13:03.259300 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259262    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259555    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1972c32-1497-43e9-9837-279d28a0b6ac" containerName="loki-distributor"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259566    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1972c32-1497-43e9-9837-279d28a0b6ac" containerName="loki-distributor"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259576    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" containerName="loki-compactor"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259581    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" containerName="loki-compactor"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259589    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="opa"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259595    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="opa"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259604    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="gateway"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259610    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="gateway"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259617    4528 cpu_manager.go:414] "containerMap: removing container" podUID="76e3de77-a015-46f7-b970-a14b9d6451cf" containerName="otc-container"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259622    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e3de77-a015-46f7-b970-a14b9d6451cf" containerName="otc-container"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259638    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester"
Oct 30 04:13:03.259656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259643    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259690    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee80cd6-0004-4966-8ad0-2f6e9fecc51d" containerName="loki-ingester"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259698    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1972c32-1497-43e9-9837-279d28a0b6ac" containerName="loki-distributor"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259704    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a66854b-8efa-43d6-aa8d-8b3f2b499e57" containerName="loki-compactor"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259710    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="opa"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259717    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ff0038-854c-4145-948e-5f794ba65fcd" containerName="gateway"
Oct 30 04:13:03.260166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.259723    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e3de77-a015-46f7-b970-a14b9d6451cf" containerName="otc-container"
Oct 30 04:13:03.262618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.262600    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.292608 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.292570    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:13:03.296032 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.295999    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:03.326319 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.326283    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.326468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.326388    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.326514 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.326473    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99q2l\" (UniqueName: \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.326514 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.326499    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.326599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.326578    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.353047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353012    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-http\""
Oct 30 04:13:03.353189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353129    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-vft9z\""
Oct 30 04:13:03.353189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353148    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-distributor-grpc\""
Oct 30 04:13:03.353599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353586    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:13:03.353734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353721    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\""
Oct 30 04:13:03.353775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.353739    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:13:03.364591 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.364563    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\""
Oct 30 04:13:03.427973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.427932    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.427981    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.428097    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99q2l\" (UniqueName: \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.428133    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428350 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.428173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.428811    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.428969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.428908    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.430592 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.430568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.430863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.430839    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.433727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.433702    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:03.447647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.447620    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:13:03.450861 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.450842    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.454539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.454514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99q2l\" (UniqueName: \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\") pod \"data-science-lokistack-distributor-58947679b7-7d8r8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") " pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.474863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.474829    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:13:03.529258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.529258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529213    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77gxv\" (UniqueName: \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.529258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529236    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.529545 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.529545 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529466    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.529545 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.529501    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.557468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.557430    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\""
Oct 30 04:13:03.557732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.557681    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\""
Oct 30 04:13:03.557882 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.557868    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\""
Oct 30 04:13:03.573942 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.573915    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:03.575484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.575449    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630309    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630543    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630631    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630695    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.631921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.630721    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77gxv\" (UniqueName: \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.632293 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.632037    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.632713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.632687    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.635432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.635305    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.635432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.635328    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.636468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.636444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.658016 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.657986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77gxv\" (UniqueName: \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\") pod \"data-science-lokistack-querier-56ffbd5fdc-kll2c\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.710898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.710863    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:13:03.715313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.715295    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.758539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.758505    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:13:03.769352 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.769328    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:03.779723 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.779690    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:03.780798 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.780763    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-grpc\""
Oct 30 04:13:03.780898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.780805    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-index-gateway-http\""
Oct 30 04:13:03.885463 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.885434    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:13:03.887910 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:13:03.887878    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43354ee_6cde_461b_b3aa_28a882161aa8.slice/crio-6baa286d300785ea1151471d77454f19a8bdab02f3d84334b1a768b0dea30fe3 WatchSource:0}: Error finding container 6baa286d300785ea1151471d77454f19a8bdab02f3d84334b1a768b0dea30fe3: Status 404 returned error can't find the container with id 6baa286d300785ea1151471d77454f19a8bdab02f3d84334b1a768b0dea30fe3
Oct 30 04:13:03.910456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.910417    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:13:03.916943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.916908    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:03.920904 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.920785    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:03.934105 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.934235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.934235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934128    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxc77\" (UniqueName: \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.934235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934216    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.934424 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934339    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.934424 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.934382    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:03.936372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.936351    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:13:03.938562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.938523    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\""
Oct 30 04:13:03.940099 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.939082    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:13:03.941328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.941303    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\""
Oct 30 04:13:03.941328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.941321    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\""
Oct 30 04:13:03.941470 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:03.941432    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\""
Oct 30 04:13:04.035315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035237    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035275    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035306    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035411    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035433    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035472    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035624 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035593    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035638    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035691 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035665    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxc77\" (UniqueName: \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035693    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dt5\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035736    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.035878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035764    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035790    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.035878 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.035866    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.036311 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.036266    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.036444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.036354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.038072 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.038048    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.038199 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.038090    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.038632 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.038611    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.049396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.049367    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:04.077673 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.077642    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxc77\" (UniqueName: \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:04.082290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.082262    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:13:04.084775 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:13:04.084748    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d70aeb_a448_48e8_a3c5_f02413f063e2.slice/crio-e8fdd2fcf419f7a9582a737841544f9eb639f16a2ea11cffac11f9692f0d6899 WatchSource:0}: Error finding container e8fdd2fcf419f7a9582a737841544f9eb639f16a2ea11cffac11f9692f0d6899: Status 404 returned error can't find the container with id e8fdd2fcf419f7a9582a737841544f9eb639f16a2ea11cffac11f9692f0d6899
Oct 30 04:13:04.136262 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136279    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136306    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dt5\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136355    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136391    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136668 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136421    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.136668 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.136452    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.137137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.137074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.137261 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.137214    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.137403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.137384    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.139095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.139070    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.139226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.139203    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.139343 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.139325    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.139384 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.139341    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.160792 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.160762    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dt5\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\") pod \"data-science-lokistack-gateway-557958b888-lnj5p\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.176793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.176755    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:04.235625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.235585    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:04.377500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.377463    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" event={"ID":"b8d70aeb-a448-48e8-a3c5-f02413f063e2","Type":"ContainerStarted","Data":"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"}
Oct 30 04:13:04.377974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.377534    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:04.377974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.377558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" event={"ID":"b8d70aeb-a448-48e8-a3c5-f02413f063e2","Type":"ContainerStarted","Data":"e8fdd2fcf419f7a9582a737841544f9eb639f16a2ea11cffac11f9692f0d6899"}
Oct 30 04:13:04.379012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.378988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" event={"ID":"f43354ee-6cde-461b-b3aa-28a882161aa8","Type":"ContainerStarted","Data":"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"}
Oct 30 04:13:04.379118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.379033    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" event={"ID":"f43354ee-6cde-461b-b3aa-28a882161aa8","Type":"ContainerStarted","Data":"6baa286d300785ea1151471d77454f19a8bdab02f3d84334b1a768b0dea30fe3"}
Oct 30 04:13:04.379118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.379055    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:04.419162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.419114    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" podStartSLOduration=1.419100853 podStartE2EDuration="1.419100853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:04.418274513 +0000 UTC m=+5542.642523792" watchObservedRunningTime="2026-10-30 04:13:04.419100853 +0000 UTC m=+5542.643350129"
Oct 30 04:13:04.460241 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.460187    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" podStartSLOduration=1.460168261 podStartE2EDuration="1.460168261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:04.458450272 +0000 UTC m=+5542.682699563" watchObservedRunningTime="2026-10-30 04:13:04.460168261 +0000 UTC m=+5542.684417539"
Oct 30 04:13:04.559759 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:04.559731    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:13:04.562363 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:13:04.562335    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925760fb_bd79_44ad_ae41_6a0f351d904f.slice/crio-13653707ec1cc2eba837ab9a0db2526b07e97c2e5fb28530822083cbac9b6ee8 WatchSource:0}: Error finding container 13653707ec1cc2eba837ab9a0db2526b07e97c2e5fb28530822083cbac9b6ee8: Status 404 returned error can't find the container with id 13653707ec1cc2eba837ab9a0db2526b07e97c2e5fb28530822083cbac9b6ee8
Oct 30 04:13:04.633033 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:13:04.632994    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:13:05.384921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.384884    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerStarted","Data":"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"}
Oct 30 04:13:05.385509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.384949    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerStarted","Data":"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"}
Oct 30 04:13:05.385509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.384971    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerStarted","Data":"13653707ec1cc2eba837ab9a0db2526b07e97c2e5fb28530822083cbac9b6ee8"}
Oct 30 04:13:05.385509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.385302    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:05.391625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.391600    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:05.428920 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:05.428863    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" podStartSLOduration=2.42884612 podStartE2EDuration="2.42884612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:05.427626501 +0000 UTC m=+5543.651875791" watchObservedRunningTime="2026-10-30 04:13:05.42884612 +0000 UTC m=+5543.653095398"
Oct 30 04:13:06.390385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:06.390353    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:06.398369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:06.398344    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:13:07.162405 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.162368    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.190242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.190212    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.263472 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.263431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.263622 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.263555    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.277806 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.277783    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") DevicePath \"csi-9eab6a07b1d61d94cb8047377e7bdc52265c941fe4e8c4cb4476f1cdd006914d\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.381414 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.381386    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d0ba80d454beb69db9470fc45e5a47dc3c954c5997bec8b3d02403b351bf919d/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.406410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.406377    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"data-science-lokistack-index-gateway-0\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") " pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.626785 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.626704    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:07.924297 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:07.924261    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:13:07.927011 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:13:07.926985    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b248b3_86a6_4158_883d_c711171d7f72.slice/crio-6ff71895cf50192fda2b7286c732f0ec4b6e571c1060b54b62cc070a5ba1d852 WatchSource:0}: Error finding container 6ff71895cf50192fda2b7286c732f0ec4b6e571c1060b54b62cc070a5ba1d852: Status 404 returned error can't find the container with id 6ff71895cf50192fda2b7286c732f0ec4b6e571c1060b54b62cc070a5ba1d852
Oct 30 04:13:08.399916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:08.399882    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f2b248b3-86a6-4158-883d-c711171d7f72","Type":"ContainerStarted","Data":"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"}
Oct 30 04:13:08.400076 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:08.399933    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f2b248b3-86a6-4158-883d-c711171d7f72","Type":"ContainerStarted","Data":"6ff71895cf50192fda2b7286c732f0ec4b6e571c1060b54b62cc070a5ba1d852"}
Oct 30 04:13:09.404617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:09.404584    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:25.392113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:25.392083    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:13:25.392517 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:25.392185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:13:25.433983 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:25.433933    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podStartSLOduration=22.433919005 podStartE2EDuration="22.433919005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:13:08.443549122 +0000 UTC m=+5546.667798400" watchObservedRunningTime="2026-10-30 04:13:25.433919005 +0000 UTC m=+5563.658168282"
Oct 30 04:13:25.742070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:25.741977    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:30.414761 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:30.414731    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:13:34.630578 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:13:34.630540    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:13:41.159430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:41.159391    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:13:41.291086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:13:41.291044    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:04.631553 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:14:04.631510    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:14:14.718305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.718271    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:14:14.721524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.721497    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.747417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.747381    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-http\""
Oct 30 04:14:14.747576 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.747423    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ingester-grpc\""
Oct 30 04:14:14.748901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.748876    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:14:14.959499 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959469    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.959499 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959502    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhfdv\" (UniqueName: \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.959727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959524    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.959727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959642    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.959727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959678    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:14.959929 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:14.959840    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063393    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063490 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063481    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063498    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhfdv\" (UniqueName: \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063523    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063550    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.063751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.063573    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.064468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.064437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.064672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.064640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.066553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.066509    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.066691 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.066556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.067519 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.067496    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:15.090376 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:15.090346    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhfdv\" (UniqueName: \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.899086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.899042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.899086 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.899094    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.927158 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.927123    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.927262 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.927186    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.999650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.999615    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.999650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.999657    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.999943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.999756    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:18.999943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:18.999775    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.013611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.013582    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") DevicePath \"csi-513b28985aab521b4eb82938c6eae5a98ecd747b2f9bb9f4846be96113110cfb\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.013611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.013604    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") DevicePath \"csi-69a6f96e0c254e55072bf9442d149758a25a7d3ae16dfffd19da77a5e7e0b8c8\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.120454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.120424    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fcf6f0536bbc97a5651ec9a8d066f70948542721519078012a300fe6ca3edfd0/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.131005 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.130973    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/be8d3d2716806c45d69e88ad453cb51178b1b3855120bf6702b7eff86c093a7e/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.150134 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.150073    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.153843 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.153809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"data-science-lokistack-ingester-1\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") " pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.235381 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.235345    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.548301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.548269    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:14:19.550717 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:14:19.550692    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefae04a_c8c1_4b4f_9620_f393ad6e385a.slice/crio-4aef13b6dd4b8f1dc1acbf7f6ff44ec27b44ebd71be34ce2bf93164ada5da5cc WatchSource:0}: Error finding container 4aef13b6dd4b8f1dc1acbf7f6ff44ec27b44ebd71be34ce2bf93164ada5da5cc: Status 404 returned error can't find the container with id 4aef13b6dd4b8f1dc1acbf7f6ff44ec27b44ebd71be34ce2bf93164ada5da5cc
Oct 30 04:14:19.700716 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.700674    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"befae04a-c8c1-4b4f-9620-f393ad6e385a","Type":"ContainerStarted","Data":"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"}
Oct 30 04:14:19.700914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.700734    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"befae04a-c8c1-4b4f-9620-f393ad6e385a","Type":"ContainerStarted","Data":"4aef13b6dd4b8f1dc1acbf7f6ff44ec27b44ebd71be34ce2bf93164ada5da5cc"}
Oct 30 04:14:19.700914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.700785    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:14:19.743352 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:19.743296    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podStartSLOduration=5.743278306 podStartE2EDuration="5.743278306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:19.742619821 +0000 UTC m=+5617.966869099" watchObservedRunningTime="2026-10-30 04:14:19.743278306 +0000 UTC m=+5617.967527583"
Oct 30 04:14:34.631184 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:14:34.631137    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:14:40.710928 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:40.710889    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:14:41.268438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:41.268400    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:14:50.708941 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:14:50.708902    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:15:00.708882 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:15:00.708848    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:15:04.631155 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:15:04.631119    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:15:25.199468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:15:25.199430    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:26.665678 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:15:26.665638    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:15:34.631684 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:15:34.631645    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:15:41.459273 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:15:41.459226    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:04.630467 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:04.630391    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:16:12.322579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.322541    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:12.327637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.327597    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.341313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.341288    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44tt\" (UniqueName: \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.341441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.341331    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.357579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.357539    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:12.357713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.357682    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:12.376344 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.376319    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-j5fn8\""
Oct 30 04:16:12.416604 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.416576    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-6a6878d3\""
Oct 30 04:16:12.442088 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.442061    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44tt\" (UniqueName: \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.442208 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.442106    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.442679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.442661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.465277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.465248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44tt\" (UniqueName: \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\") pod \"data-science-usage-logs-collector-5d66b649d7-pnhjh\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.490588 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.490559    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:12.641547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.641519    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:12.938040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.938011    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:12.940574 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:16:12.940546    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0bb9d6_8e25_40e4_9a1b_1d19fd9387c1.slice/crio-b6f86c0797d0754c98c2aa61550e40502f7b08a7c452caaca94e200295bade31 WatchSource:0}: Error finding container b6f86c0797d0754c98c2aa61550e40502f7b08a7c452caaca94e200295bade31: Status 404 returned error can't find the container with id b6f86c0797d0754c98c2aa61550e40502f7b08a7c452caaca94e200295bade31
Oct 30 04:16:12.942687 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.942656    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:16:12.942771 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:12.942713    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:16:13.149003 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:13.148961    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" event={"ID":"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1","Type":"ContainerStarted","Data":"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"}
Oct 30 04:16:13.149177 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:13.149024    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" event={"ID":"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1","Type":"ContainerStarted","Data":"b6f86c0797d0754c98c2aa61550e40502f7b08a7c452caaca94e200295bade31"}
Oct 30 04:16:13.190385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:13.190292    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" podStartSLOduration=1.190269493 podStartE2EDuration="1.190269493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:13.188616904 +0000 UTC m=+5731.412866180" watchObservedRunningTime="2026-10-30 04:16:13.190269493 +0000 UTC m=+5731.414518773"
Oct 30 04:16:16.158184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.158145    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:16.163095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.163061    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:16:16.163314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.163290    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" podUID="f2b248b3-86a6-4158-883d-c711171d7f72" containerName="loki-index-gateway" containerID="cri-o://d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9" gracePeriod=30
Oct 30 04:16:16.203121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.203066    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:16:16.203315 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.203282    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester" containerID="cri-o://d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02" gracePeriod=600
Oct 30 04:16:16.214919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.214891    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:16:16.215117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.215094    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor" containerID="cri-o://93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62" gracePeriod=30
Oct 30 04:16:16.222531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.222499    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:16:16.222730 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.222705    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" podUID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" containerName="loki-querier" containerID="cri-o://027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965" gracePeriod=30
Oct 30 04:16:16.235454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.235420    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:16:16.236123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.235661    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="gateway" containerID="cri-o://5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80" gracePeriod=30
Oct 30 04:16:16.236123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.235871    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="opa" containerID="cri-o://20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c" gracePeriod=30
Oct 30 04:16:16.350756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.350709    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:16.382421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.382385    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:16.382614 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.382554    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" podUID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" containerName="otc-container" containerID="cri-o://75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74" gracePeriod=30
Oct 30 04:16:16.395949 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.395870    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:16:16.495054 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.495011    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:16.561716 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.556003    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579387    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\" (UniqueName: \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579523    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579565    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579626    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.579673    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\") pod \"f2b248b3-86a6-4158-883d-c711171d7f72\" (UID: \"f2b248b3-86a6-4158-883d-c711171d7f72\") "
Oct 30 04:16:16.580688 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.580267    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.580688 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.580397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.582453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.582366    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "kube-api-access-vxc77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:16.582453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.582378    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "data-science-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.582453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.582429    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "data-science-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.582941 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.582918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "pvc-536701e3-6ae5-4168-ad78-e30a293330df". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:16.583938 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.583914    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret" pod "f2b248b3-86a6-4158-883d-c711171d7f72" (UID: "f2b248b3-86a6-4158-883d-c711171d7f72"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.680791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680760    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.680791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680791    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-index-gateway-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.681066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680804    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.681066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680843    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxc77\" (UniqueName: \"kubernetes.io/projected/f2b248b3-86a6-4158-883d-c711171d7f72-kube-api-access-vxc77\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.681066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680883    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:16:16.681066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680897    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2b248b3-86a6-4158-883d-c711171d7f72-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.681066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.680909    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/f2b248b3-86a6-4158-883d-c711171d7f72-test-loki-lifecycle-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.690721 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.690667    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-536701e3-6ae5-4168-ad78-e30a293330df" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:16:16.713440 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.713417    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:16.730094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.730074    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:16:16.781192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781166    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781355 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781197    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781355 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781227    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781355 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781266    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\" (UniqueName: \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\") pod \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") "
Oct 30 04:16:16.781355 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781326    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\") pod \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\" (UID: \"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1\") "
Oct 30 04:16:16.781590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\" (UniqueName: \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781482    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\") pod \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\" (UID: \"b8d70aeb-a448-48e8-a3c5-f02413f063e2\") "
Oct 30 04:16:16.781857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781755    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-536701e3-6ae5-4168-ad78-e30a293330df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-075058d7d916c118a\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-9eab6a07b1d61d94cb8047377e7bdc52265c941fe4e8c4cb4476f1cdd006914d\""
Oct 30 04:16:16.781857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781779    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal" pod "6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" (UID: "6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.781857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.781788    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.782130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.782106    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:16.783593 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.783564    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.783694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.783586    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.783752 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.783722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:16.783913 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.783899    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt" pod "6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" (UID: "6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1"). InnerVolumeSpecName "kube-api-access-p44tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:16.784192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.784170    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv" pod "b8d70aeb-a448-48e8-a3c5-f02413f063e2" (UID: "b8d70aeb-a448-48e8-a3c5-f02413f063e2"). InnerVolumeSpecName "kube-api-access-77gxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:16.882270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882218    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77gxv\" (UniqueName: \"kubernetes.io/projected/b8d70aeb-a448-48e8-a3c5-f02413f063e2-kube-api-access-77gxv\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882262    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882273    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-test-loki-lifecycle-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882282    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882291    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/b8d70aeb-a448-48e8-a3c5-f02413f063e2-data-science-lokistack-querier-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882302    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44tt\" (UniqueName: \"kubernetes.io/projected/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-kube-api-access-p44tt\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882310    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:16.882546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:16.882319    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b8d70aeb-a448-48e8-a3c5-f02413f063e2-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.139938 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.139911    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:16:17.169530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.169503    4528 generic.go:356] "Generic (PLEG): container finished" podID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerID="20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c" exitCode=0
Oct 30 04:16:17.169952 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.169574    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerDied","Data":"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"}
Oct 30 04:16:17.170852 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.170834    4528 generic.go:356] "Generic (PLEG): container finished" podID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerID="d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02" exitCode=0
Oct 30 04:16:17.170947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.170871    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1"
Oct 30 04:16:17.170947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.170905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"befae04a-c8c1-4b4f-9620-f393ad6e385a","Type":"ContainerDied","Data":"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"}
Oct 30 04:16:17.171044 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.170950    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-ingester-1" event={"ID":"befae04a-c8c1-4b4f-9620-f393ad6e385a","Type":"ContainerDied","Data":"4aef13b6dd4b8f1dc1acbf7f6ff44ec27b44ebd71be34ce2bf93164ada5da5cc"}
Oct 30 04:16:17.171044 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.170970    4528 scope.go:122] "RemoveContainer" containerID="d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"
Oct 30 04:16:17.172190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.172146    4528 generic.go:356] "Generic (PLEG): container finished" podID="f2b248b3-86a6-4158-883d-c711171d7f72" containerID="d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9" exitCode=0
Oct 30 04:16:17.172255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.172192    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0"
Oct 30 04:16:17.172255 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.172227    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f2b248b3-86a6-4158-883d-c711171d7f72","Type":"ContainerDied","Data":"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"}
Oct 30 04:16:17.172374 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.172270    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-index-gateway-0" event={"ID":"f2b248b3-86a6-4158-883d-c711171d7f72","Type":"ContainerDied","Data":"6ff71895cf50192fda2b7286c732f0ec4b6e571c1060b54b62cc070a5ba1d852"}
Oct 30 04:16:17.173707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.173675    4528 generic.go:356] "Generic (PLEG): container finished" podID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" containerID="75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74" exitCode=0
Oct 30 04:16:17.173829 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.173712    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"
Oct 30 04:16:17.173829 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.173756    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" event={"ID":"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1","Type":"ContainerDied","Data":"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"}
Oct 30 04:16:17.173829 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.173795    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh" event={"ID":"6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1","Type":"ContainerDied","Data":"b6f86c0797d0754c98c2aa61550e40502f7b08a7c452caaca94e200295bade31"}
Oct 30 04:16:17.175232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.175209    4528 generic.go:356] "Generic (PLEG): container finished" podID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" containerID="027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965" exitCode=0
Oct 30 04:16:17.175341 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.175257    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"
Oct 30 04:16:17.175341 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.175259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" event={"ID":"b8d70aeb-a448-48e8-a3c5-f02413f063e2","Type":"ContainerDied","Data":"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"}
Oct 30 04:16:17.175442 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.175361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c" event={"ID":"b8d70aeb-a448-48e8-a3c5-f02413f063e2","Type":"ContainerDied","Data":"e8fdd2fcf419f7a9582a737841544f9eb639f16a2ea11cffac11f9692f0d6899"}
Oct 30 04:16:17.185506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.185483    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.185609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.185592    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.185673 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.185658    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.185756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.185738    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.185957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.185939    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.192066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.191285    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.192066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.191361    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.192066 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.191438    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\" (UniqueName: \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\") pod \"befae04a-c8c1-4b4f-9620-f393ad6e385a\" (UID: \"befae04a-c8c1-4b4f-9620-f393ad6e385a\") "
Oct 30 04:16:17.196277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.190290    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "data-science-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:17.196382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.190524    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:17.196382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.190976    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:17.196382 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.196242    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "data-science-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:17.196737 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.196397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "kube-api-access-dhfdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:17.196795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.196750    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:17.196795 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.196775    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "pvc-199f9790-eb98-4bad-8977-a99a3b174eed". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:16:17.196925 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.196889    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle" pod "befae04a-c8c1-4b4f-9620-f393ad6e385a" (UID: "befae04a-c8c1-4b4f-9620-f393ad6e385a"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:17.216058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.216033    4528 scope.go:122] "RemoveContainer" containerID="d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"
Oct 30 04:16:17.216423 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:17.216389    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02\": container with ID starting with d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02 not found: ID does not exist" containerID="d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"
Oct 30 04:16:17.216520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.216427    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02"} err="failed to get container status \"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02\": rpc error: code = NotFound desc = could not find container \"d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02\": container with ID starting with d3522c30165b5ecfece3701f614d80a9c655689a675dc2d6897b592de46e1d02 not found: ID does not exist"
Oct 30 04:16:17.216520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.216450    4528 scope.go:122] "RemoveContainer" containerID="d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"
Oct 30 04:16:17.224845 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.224794    4528 scope.go:122] "RemoveContainer" containerID="d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"
Oct 30 04:16:17.225127 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:17.225108    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9\": container with ID starting with d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9 not found: ID does not exist" containerID="d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"
Oct 30 04:16:17.225219 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.225137    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9"} err="failed to get container status \"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9\": rpc error: code = NotFound desc = could not find container \"d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9\": container with ID starting with d6d848fdfe6140bde012e47d42142fd513c52b1f7a2591491dd689b35aedead9 not found: ID does not exist"
Oct 30 04:16:17.225219 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.225160    4528 scope.go:122] "RemoveContainer" containerID="75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"
Oct 30 04:16:17.232454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.232437    4528 scope.go:122] "RemoveContainer" containerID="75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"
Oct 30 04:16:17.232735 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:17.232712    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74\": container with ID starting with 75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74 not found: ID does not exist" containerID="75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"
Oct 30 04:16:17.232897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.232740    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74"} err="failed to get container status \"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74\": rpc error: code = NotFound desc = could not find container \"75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74\": container with ID starting with 75a8c77e97a7eb870e51013e8e33d962af7259b1f9e08f88213c7d23f4c09b74 not found: ID does not exist"
Oct 30 04:16:17.232897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.232756    4528 scope.go:122] "RemoveContainer" containerID="027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"
Oct 30 04:16:17.240175 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.240158    4528 scope.go:122] "RemoveContainer" containerID="027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"
Oct 30 04:16:17.240417 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:17.240399    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965\": container with ID starting with 027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965 not found: ID does not exist" containerID="027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"
Oct 30 04:16:17.240465 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.240424    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965"} err="failed to get container status \"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965\": rpc error: code = NotFound desc = could not find container \"027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965\": container with ID starting with 027031ade2adedf847efa608cccdd06ddb98f5eaeb9341857494a6d8110a6965 not found: ID does not exist"
Oct 30 04:16:17.281947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.281870    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:16:17.288214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.288188    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-56ffbd5fdc-kll2c"]
Oct 30 04:16:17.295226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295194    4528 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-test-loki-lifecycle-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.295290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295248    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:16:17.295290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295265    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.295290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295280    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:16:17.295290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295290    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.295438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295300    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ingester-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.295438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295313    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befae04a-c8c1-4b4f-9620-f393ad6e385a-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.295438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.295326    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhfdv\" (UniqueName: \"kubernetes.io/projected/befae04a-c8c1-4b4f-9620-f393ad6e385a-kube-api-access-dhfdv\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.305468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.305448    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-199f9790-eb98-4bad-8977-a99a3b174eed" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:16:17.309101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.309084    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-ea00ca9c-db2e-4885-bfa1-004970abec93" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:16:17.321565 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.321544    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:17.326260 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.326241    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5d66b649d7-pnhjh"]
Oct 30 04:16:17.358414 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.358390    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:16:17.362932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.362906    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-index-gateway-0"]
Oct 30 04:16:17.396205 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.396164    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-199f9790-eb98-4bad-8977-a99a3b174eed\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-057918a7bba054950\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-513b28985aab521b4eb82938c6eae5a98ecd747b2f9bb9f4846be96113110cfb\""
Oct 30 04:16:17.396205 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.396194    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-ea00ca9c-db2e-4885-bfa1-004970abec93\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0afa2a60e04c19735\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-69a6f96e0c254e55072bf9442d149758a25a7d3ae16dfffd19da77a5e7e0b8c8\""
Oct 30 04:16:17.484017 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.483993    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:16:17.516354 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.516323    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:16:17.520113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.520090    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-ingester-1"]
Oct 30 04:16:17.597799 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597715    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.597799 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597771    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597849    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597929    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.597977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.598014    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.598049    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\") pod \"925760fb-bd79-44ad-ae41-6a0f351d904f\" (UID: \"925760fb-bd79-44ad-ae41-6a0f351d904f\") "
Oct 30 04:16:17.598323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.598212    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:17.598431 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.598354    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.600416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.600389    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "kube-api-access-89dt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:17.600536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.600491    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:17.600664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.600646    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:17.600746 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.600658    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:17.602857 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.602836    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:17.604897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.604877    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:17.605978 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.605960    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants" pod "925760fb-bd79-44ad-ae41-6a0f351d904f" (UID: "925760fb-bd79-44ad-ae41-6a0f351d904f"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:17.699036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699004    4528 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-tls-secret\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699036 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699033    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699044    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dt5\" (UniqueName: \"kubernetes.io/projected/925760fb-bd79-44ad-ae41-6a0f351d904f-kube-api-access-89dt5\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699055    4528 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-lokistack-gateway\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699065    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699074    4528 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/925760fb-bd79-44ad-ae41-6a0f351d904f-tenants\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.699242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.699082    4528 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/925760fb-bd79-44ad-ae41-6a0f351d904f-rbac\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:17.891853 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.891809    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" path="/var/lib/kubelet/pods/6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1/volumes"
Oct 30 04:16:17.892216 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.892203    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" path="/var/lib/kubelet/pods/b8d70aeb-a448-48e8-a3c5-f02413f063e2/volumes"
Oct 30 04:16:17.892566 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.892555    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" path="/var/lib/kubelet/pods/befae04a-c8c1-4b4f-9620-f393ad6e385a/volumes"
Oct 30 04:16:17.893247 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:17.893232    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b248b3-86a6-4158-883d-c711171d7f72" path="/var/lib/kubelet/pods/f2b248b3-86a6-4158-883d-c711171d7f72/volumes"
Oct 30 04:16:18.183033 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.183001    4528 generic.go:356] "Generic (PLEG): container finished" podID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerID="5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80" exitCode=0
Oct 30 04:16:18.183539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.183064    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"
Oct 30 04:16:18.183539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.183103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerDied","Data":"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"}
Oct 30 04:16:18.183539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.183190    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p" event={"ID":"925760fb-bd79-44ad-ae41-6a0f351d904f","Type":"ContainerDied","Data":"13653707ec1cc2eba837ab9a0db2526b07e97c2e5fb28530822083cbac9b6ee8"}
Oct 30 04:16:18.183539 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.183224    4528 scope.go:122] "RemoveContainer" containerID="20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"
Oct 30 04:16:18.192743 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.192726    4528 scope.go:122] "RemoveContainer" containerID="5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"
Oct 30 04:16:18.200448 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.200432    4528 scope.go:122] "RemoveContainer" containerID="20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"
Oct 30 04:16:18.200722 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:18.200702    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c\": container with ID starting with 20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c not found: ID does not exist" containerID="20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"
Oct 30 04:16:18.200772 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.200731    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c"} err="failed to get container status \"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c\": rpc error: code = NotFound desc = could not find container \"20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c\": container with ID starting with 20f35261e41884e9ad19464078b6d09afc0ebe1dcde92a75bf869c87e5445d8c not found: ID does not exist"
Oct 30 04:16:18.200772 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.200747    4528 scope.go:122] "RemoveContainer" containerID="5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"
Oct 30 04:16:18.201011 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:18.200993    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80\": container with ID starting with 5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80 not found: ID does not exist" containerID="5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"
Oct 30 04:16:18.201057 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.201020    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80"} err="failed to get container status \"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80\": rpc error: code = NotFound desc = could not find container \"5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80\": container with ID starting with 5211993e311cf9ce874ac29ab1b1134ea2b4d2ae3151e0efdd0634bc973f9c80 not found: ID does not exist"
Oct 30 04:16:18.232713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.232693    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:16:18.244455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:18.244436    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-557958b888-lnj5p"]
Oct 30 04:16:19.891344 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:19.891312    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" path="/var/lib/kubelet/pods/925760fb-bd79-44ad-ae41-6a0f351d904f/volumes"
Oct 30 04:16:25.389267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:25.389225    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:16:34.632028 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:34.631985    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:16:35.389214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:35.389167    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:16:36.967957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:36.967930    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:16:37.052376 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052296    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\") pod \"f43354ee-6cde-461b-b3aa-28a882161aa8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") "
Oct 30 04:16:37.052376 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\") pod \"f43354ee-6cde-461b-b3aa-28a882161aa8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") "
Oct 30 04:16:37.052376 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\") pod \"f43354ee-6cde-461b-b3aa-28a882161aa8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") "
Oct 30 04:16:37.052683 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052391    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\") pod \"f43354ee-6cde-461b-b3aa-28a882161aa8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") "
Oct 30 04:16:37.052683 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\" (UniqueName: \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\") pod \"f43354ee-6cde-461b-b3aa-28a882161aa8\" (UID: \"f43354ee-6cde-461b-b3aa-28a882161aa8\") "
Oct 30 04:16:37.052806 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052766    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config" pod "f43354ee-6cde-461b-b3aa-28a882161aa8" (UID: "f43354ee-6cde-461b-b3aa-28a882161aa8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:37.052890 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.052808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle" pod "f43354ee-6cde-461b-b3aa-28a882161aa8" (UID: "f43354ee-6cde-461b-b3aa-28a882161aa8"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:37.054761 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.054734    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http" pod "f43354ee-6cde-461b-b3aa-28a882161aa8" (UID: "f43354ee-6cde-461b-b3aa-28a882161aa8"). InnerVolumeSpecName "data-science-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:37.054982 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.054757    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l" pod "f43354ee-6cde-461b-b3aa-28a882161aa8" (UID: "f43354ee-6cde-461b-b3aa-28a882161aa8"). InnerVolumeSpecName "kube-api-access-99q2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:37.055065 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.055004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc" pod "f43354ee-6cde-461b-b3aa-28a882161aa8" (UID: "f43354ee-6cde-461b-b3aa-28a882161aa8"). InnerVolumeSpecName "data-science-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:37.153700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.153665    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99q2l\" (UniqueName: \"kubernetes.io/projected/f43354ee-6cde-461b-b3aa-28a882161aa8-kube-api-access-99q2l\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:37.153700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.153696    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:37.153700 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.153706    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-grpc\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:37.153974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.153719    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-ca-bundle\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:37.153974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.153729    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/f43354ee-6cde-461b-b3aa-28a882161aa8-data-science-lokistack-distributor-http\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:16:37.261903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.261868    4528 generic.go:356] "Generic (PLEG): container finished" podID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerID="93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62" exitCode=0
Oct 30 04:16:37.262080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.261909    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"
Oct 30 04:16:37.262080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.261955    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" event={"ID":"f43354ee-6cde-461b-b3aa-28a882161aa8","Type":"ContainerDied","Data":"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"}
Oct 30 04:16:37.262080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.262003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8" event={"ID":"f43354ee-6cde-461b-b3aa-28a882161aa8","Type":"ContainerDied","Data":"6baa286d300785ea1151471d77454f19a8bdab02f3d84334b1a768b0dea30fe3"}
Oct 30 04:16:37.262080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.262024    4528 scope.go:122] "RemoveContainer" containerID="93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"
Oct 30 04:16:37.274575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.274553    4528 scope.go:122] "RemoveContainer" containerID="93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"
Oct 30 04:16:37.274910 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:16:37.274885    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62\": container with ID starting with 93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62 not found: ID does not exist" containerID="93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"
Oct 30 04:16:37.275012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.274917    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62"} err="failed to get container status \"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62\": rpc error: code = NotFound desc = could not find container \"93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62\": container with ID starting with 93ee12fdaf8f6cb5df9d317d2b7eb788bc0e214facb4143c433848f3bac23d62 not found: ID does not exist"
Oct 30 04:16:37.311188 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.311113    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:16:37.322643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.322612    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-distributor-58947679b7-7d8r8"]
Oct 30 04:16:37.891430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:37.891397    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" path="/var/lib/kubelet/pods/f43354ee-6cde-461b-b3aa-28a882161aa8/volumes"
Oct 30 04:16:41.536104 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:41.536066    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:47.149976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:47.149934    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:49.715580 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:49.715526    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:49.891095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:49.891037    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.013629 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:50.013538    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.144040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:50.144005    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.705793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:50.705719    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.829978 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:50.829942    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:50.956871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:50.956782    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:51.658698 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.658664    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:51.658979 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.658965    4528 cpu_manager.go:414] "containerMap: removing container" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.658981    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.658999    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659008    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659019    4528 cpu_manager.go:414] "containerMap: removing container" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="opa"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659025    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="opa"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659032    4528 cpu_manager.go:414] "containerMap: removing container" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="gateway"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659037    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="gateway"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659043    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" containerName="otc-container"
Oct 30 04:16:51.659051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659047    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" containerName="otc-container"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659060    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f2b248b3-86a6-4158-883d-c711171d7f72" containerName="loki-index-gateway"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659066    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b248b3-86a6-4158-883d-c711171d7f72" containerName="loki-index-gateway"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659077    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" containerName="loki-querier"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659083    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" containerName="loki-querier"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659152    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="opa"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659163    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="925760fb-bd79-44ad-ae41-6a0f351d904f" containerName="gateway"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659181    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0bb9d6-8e25-40e4-9a1b-1d19fd9387c1" containerName="otc-container"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659194    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b248b3-86a6-4158-883d-c711171d7f72" containerName="loki-index-gateway"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659203    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d70aeb-a448-48e8-a3c5-f02413f063e2" containerName="loki-querier"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659214    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="befae04a-c8c1-4b4f-9620-f393ad6e385a" containerName="loki-ingester"
Oct 30 04:16:51.659419 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.659225    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43354ee-6cde-461b-b3aa-28a882161aa8" containerName="loki-distributor"
Oct 30 04:16:51.663963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.663935    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.681272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.681246    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:16:51.684774 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.684751    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:16:51.685094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.685076    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:16:51.694208 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.694183    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:51.696841 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.696785    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:16:51.696841 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.696828    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:16:51.697019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.696843    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\""
Oct 30 04:16:51.697019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.696894    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:16:51.725524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.725492    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:51.768849 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.768806    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.769004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.768920    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8d6\" (UniqueName: \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.769004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.768958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.769004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.768982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.769148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.769118    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.869569 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.869530    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.869594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8d6\" (UniqueName: \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.869634    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.869661    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.869712    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870582 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.870555    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.870923 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.870885    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:51.872512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.872483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.872638 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.872564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.874160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.874135    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.893530 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.893506    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8d6\" (UniqueName: \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\") pod \"data-science-collector-collector-0\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:51.979416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:51.979331    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:16:52.001677 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:52.001639    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:52.282648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:52.282621    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:16:52.284923 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:16:52.284887    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6e2547_a2d2_4783_8693_ec89d2b31cbe.slice/crio-0aebfc31412666dcbb317605b9bcc1904aa2951d6dc88966e3f1653db5a0d20f WatchSource:0}: Error finding container 0aebfc31412666dcbb317605b9bcc1904aa2951d6dc88966e3f1653db5a0d20f: Status 404 returned error can't find the container with id 0aebfc31412666dcbb317605b9bcc1904aa2951d6dc88966e3f1653db5a0d20f
Oct 30 04:16:52.287073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:52.287039    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:16:52.287162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:52.287099    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:16:52.324389 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:52.324358    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ee6e2547-a2d2-4783-8693-ec89d2b31cbe","Type":"ContainerStarted","Data":"0aebfc31412666dcbb317605b9bcc1904aa2951d6dc88966e3f1653db5a0d20f"}
Oct 30 04:16:53.329279 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:53.329243    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ee6e2547-a2d2-4783-8693-ec89d2b31cbe","Type":"ContainerStarted","Data":"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"}
Oct 30 04:16:53.370525 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:53.370477    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.370461818 podStartE2EDuration="2.370461818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.370303279 +0000 UTC m=+5771.594552560" watchObservedRunningTime="2026-10-30 04:16:53.370461818 +0000 UTC m=+5771.594711093"
Oct 30 04:16:54.412500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.412459    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:16:54.415946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.415929    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:16:54.433977 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.433950    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:16:54.436454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.436427    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:16:54.594168 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.594131    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phx67\" (UniqueName: \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29\" (UID: \"00d7b8f1-526f-46ab-91ab-5b19904635c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:16:54.628637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.628600    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.694891 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.694801    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phx67\" (UniqueName: \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29\" (UID: \"00d7b8f1-526f-46ab-91ab-5b19904635c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:16:54.718460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.718427    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phx67\" (UniqueName: \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29\" (UID: \"00d7b8f1-526f-46ab-91ab-5b19904635c4\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:16:54.727063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.727035    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:16:54.871430 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.871396    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:54.989622 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:54.989519    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:55.020118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.020085    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:16:55.022813 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:16:55.022779    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00d7b8f1_526f_46ab_91ab_5b19904635c4.slice/crio-3172b585845a30f64b3232a0929458376da90309483a4f23557a55c2eeb26cfc WatchSource:0}: Error finding container 3172b585845a30f64b3232a0929458376da90309483a4f23557a55c2eeb26cfc: Status 404 returned error can't find the container with id 3172b585845a30f64b3232a0929458376da90309483a4f23557a55c2eeb26cfc
Oct 30 04:16:55.314111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.314020    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:55.317409 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.317384    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.322548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.322522    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:55.332786 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.332764    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:16:55.332979 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.332958    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:16:55.338642 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.338613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" event={"ID":"00d7b8f1-526f-46ab-91ab-5b19904635c4","Type":"ContainerStarted","Data":"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"}
Oct 30 04:16:55.338803 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.338651    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" event={"ID":"00d7b8f1-526f-46ab-91ab-5b19904635c4","Type":"ContainerStarted","Data":"3172b585845a30f64b3232a0929458376da90309483a4f23557a55c2eeb26cfc"}
Oct 30 04:16:55.346237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.345154    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:16:55.346767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.346744    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:16:55.401110 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401119    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401154    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401215    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401255    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlps8\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401283    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401305    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401487    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.401747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.401589    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.403685 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.403646    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" podStartSLOduration=1.403634587 podStartE2EDuration="1.403634587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:55.403026464 +0000 UTC m=+5773.627275743" watchObservedRunningTime="2026-10-30 04:16:55.403634587 +0000 UTC m=+5773.627883893"
Oct 30 04:16:55.418672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418647    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418694    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418841    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418895    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418904    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418904    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:16:55.419042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.418940    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-vt8ws\""
Oct 30 04:16:55.460353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.460316    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:16:55.502424 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502388    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502433    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502549    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502653    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502812 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502794    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502891 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502860    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlps8\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.502955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502935    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.503063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.502975    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.503063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.503006    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.503063 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.503032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.504494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.503939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.504494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.504433    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.505275 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.505250    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.505633 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.505605    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.505863 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.505841    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506099 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506082    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506180 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506107    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506186    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506300 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506280    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506334 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506324    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.506922 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.506907    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:16:55.526417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:16:55.526394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlps8\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.763771 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.763727    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.792000 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.791966    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.864378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.864338    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.864543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.864450    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.878472 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.878437    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:02.980724 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:02.980688    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:03.006303 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:03.006277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:03.137211 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:03.137177    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:03.468871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:03.468845    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:17:03.471281 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:17:03.471250    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165df98a_1ed0_4ede_b5a8_a45ea59f493c.slice/crio-f2711e232c230f361d8af95d933a0339e8e351301a1ccc712cb54e4c33141ab5 WatchSource:0}: Error finding container f2711e232c230f361d8af95d933a0339e8e351301a1ccc712cb54e4c33141ab5: Status 404 returned error can't find the container with id f2711e232c230f361d8af95d933a0339e8e351301a1ccc712cb54e4c33141ab5
Oct 30 04:17:04.375890 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:04.375844    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerStarted","Data":"9f7c0e0a68466febebc3cd9aab23f7a74afcda75e8f40664fde1e7c00503a783"}
Oct 30 04:17:04.376401 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:04.375916    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerStarted","Data":"f2711e232c230f361d8af95d933a0339e8e351301a1ccc712cb54e4c33141ab5"}
Oct 30 04:17:04.630649 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:17:04.630616    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:17:08.392227 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:08.392189    4528 generic.go:356] "Generic (PLEG): container finished" podID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerID="9f7c0e0a68466febebc3cd9aab23f7a74afcda75e8f40664fde1e7c00503a783" exitCode=0
Oct 30 04:17:08.392696 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:08.392243    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerDied","Data":"9f7c0e0a68466febebc3cd9aab23f7a74afcda75e8f40664fde1e7c00503a783"}
Oct 30 04:17:09.398481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:09.398443    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerStarted","Data":"29632cd4c11e675150ff47d346f643390a8677c3b9b9eb7782af8cd9387701cc"}
Oct 30 04:17:09.399083 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:09.398521    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerStarted","Data":"6529f5fe18f133004538ba6e9b795105328a4dca6eb2289d3928da949302e61d"}
Oct 30 04:17:09.399083 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:09.398558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerStarted","Data":"842c58c64aac0ae8b1fe3a540b2b2c8deb0f99f3137e25660bab6e58c8d85e95"}
Oct 30 04:17:09.483015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:09.482968    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.895569273 podStartE2EDuration="14.482954689s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.587385416s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:09.471318983 +0000 UTC m=+5787.695568262" watchObservedRunningTime="2026-10-30 04:17:09.482954689 +0000 UTC m=+5787.707203965"
Oct 30 04:17:13.137567 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:13.137531    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:18.138280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:18.138240    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:18.143791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:18.143769    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:18.439666 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:18.439591    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:17:25.181325 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:25.181208    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:25.382117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:25.382082    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:25.501226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:25.501152    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:27.180062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:27.180022    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:27.301443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:27.301407    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:27.418918 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:27.418879    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:31.875608 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:31.875575    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:31.876118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:31.875832    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" containerName="otc-container" containerID="cri-o://fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920" gracePeriod=30
Oct 30 04:17:32.118052 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.118027    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.222838 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.222780    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\") pod \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") "
Oct 30 04:17:32.223055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.222862    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\") pod \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") "
Oct 30 04:17:32.223055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.222960    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\") pod \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") "
Oct 30 04:17:32.223055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.223027    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\" (UniqueName: \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\") pod \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") "
Oct 30 04:17:32.223258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.223060    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\") pod \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\" (UID: \"ee6e2547-a2d2-4783-8693-ec89d2b31cbe\") "
Oct 30 04:17:32.223258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.223240    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal" pod "ee6e2547-a2d2-4783-8693-ec89d2b31cbe" (UID: "ee6e2547-a2d2-4783-8693-ec89d2b31cbe"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:32.223391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.223375    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:32.225231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.225201    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6" pod "ee6e2547-a2d2-4783-8693-ec89d2b31cbe" (UID: "ee6e2547-a2d2-4783-8693-ec89d2b31cbe"). InnerVolumeSpecName "kube-api-access-zt8d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:32.225356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.225211    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert" pod "ee6e2547-a2d2-4783-8693-ec89d2b31cbe" (UID: "ee6e2547-a2d2-4783-8693-ec89d2b31cbe"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:32.225356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.225232    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs" pod "ee6e2547-a2d2-4783-8693-ec89d2b31cbe" (UID: "ee6e2547-a2d2-4783-8693-ec89d2b31cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:32.225356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.225247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs" pod "ee6e2547-a2d2-4783-8693-ec89d2b31cbe" (UID: "ee6e2547-a2d2-4783-8693-ec89d2b31cbe"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:32.324238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.324190    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:32.324238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.324238    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:32.324547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.324254    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt8d6\" (UniqueName: \"kubernetes.io/projected/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-kube-api-access-zt8d6\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:32.324547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.324269    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6e2547-a2d2-4783-8693-ec89d2b31cbe-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:32.492995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.492903    4528 generic.go:356] "Generic (PLEG): container finished" podID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" containerID="fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920" exitCode=0
Oct 30 04:17:32.492995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.492968    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ee6e2547-a2d2-4783-8693-ec89d2b31cbe","Type":"ContainerDied","Data":"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"}
Oct 30 04:17:32.492995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.492976    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.493274 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.493043    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ee6e2547-a2d2-4783-8693-ec89d2b31cbe","Type":"ContainerDied","Data":"0aebfc31412666dcbb317605b9bcc1904aa2951d6dc88966e3f1653db5a0d20f"}
Oct 30 04:17:32.493274 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.493066    4528 scope.go:122] "RemoveContainer" containerID="fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"
Oct 30 04:17:32.501889 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.501869    4528 scope.go:122] "RemoveContainer" containerID="fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"
Oct 30 04:17:32.502152 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:17:32.502127    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920\": container with ID starting with fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920 not found: ID does not exist" containerID="fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"
Oct 30 04:17:32.502201 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.502159    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920"} err="failed to get container status \"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920\": rpc error: code = NotFound desc = could not find container \"fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920\": container with ID starting with fbb5888cdc8c87f78b4f48ddc5269ae50c883adbb8b8a6d03871ee6fa2931920 not found: ID does not exist"
Oct 30 04:17:32.543387 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.543347    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:32.554933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.554902    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:32.585790 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.585752    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:32.586147 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.586129    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" containerName="otc-container"
Oct 30 04:17:32.586232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.586149    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" containerName="otc-container"
Oct 30 04:17:32.586232 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.586226    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" containerName="otc-container"
Oct 30 04:17:32.590305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.590283    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.605015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.604985    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:17:32.605136 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.605071    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:17:32.605238 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.605217    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:17:32.605342 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.605325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:17:32.605400 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.605348    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\""
Oct 30 04:17:32.610312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.610288    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:32.727999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.727964    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.728183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.728036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvcw\" (UniqueName: \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.728183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.728141    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.728183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.728167    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.728319 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.728192    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.828952    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.828992    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.829011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829047 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.829043    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829367 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.829086    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvcw\" (UniqueName: \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.829659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.829629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.831659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.831635    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.831659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.831652    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.831808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.831759    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.851579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.851553    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvcw\" (UniqueName: \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\") pod \"data-science-collector-collector-0\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:32.902182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:32.902153    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:33.199607 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.199573    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:33.202794 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:17:33.202769    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30108c6c_1ed0_45a7_9974_24f0e7f03c0b.slice/crio-44cca91dc9b08e16659da1fd931ae8349b18731d1ff1d15907c9f5107635d800 WatchSource:0}: Error finding container 44cca91dc9b08e16659da1fd931ae8349b18731d1ff1d15907c9f5107635d800: Status 404 returned error can't find the container with id 44cca91dc9b08e16659da1fd931ae8349b18731d1ff1d15907c9f5107635d800
Oct 30 04:17:33.205151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.205119    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:17:33.205235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.205180    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:17:33.499646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.499562    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30108c6c-1ed0-45a7-9974-24f0e7f03c0b","Type":"ContainerStarted","Data":"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"}
Oct 30 04:17:33.499646 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.499616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30108c6c-1ed0-45a7-9974-24f0e7f03c0b","Type":"ContainerStarted","Data":"44cca91dc9b08e16659da1fd931ae8349b18731d1ff1d15907c9f5107635d800"}
Oct 30 04:17:33.542136 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.542086    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.542070074 podStartE2EDuration="1.542070074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:33.541905344 +0000 UTC m=+5811.766154622" watchObservedRunningTime="2026-10-30 04:17:33.542070074 +0000 UTC m=+5811.766319346"
Oct 30 04:17:33.616309 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.616278    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:33.816531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.816441    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:33.892934 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.892901    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6e2547-a2d2-4783-8693-ec89d2b31cbe" path="/var/lib/kubelet/pods/ee6e2547-a2d2-4783-8693-ec89d2b31cbe/volumes"
Oct 30 04:17:33.927233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:33.927202    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:34.633033 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:17:34.632995    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:17:35.648396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:35.648363    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:35.764426 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:35.764388    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:36.911161 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:36.911120    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:36.911634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:36.911332    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" containerName="otc-container" containerID="cri-o://374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915" gracePeriod=30
Oct 30 04:17:37.148287 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.148262    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.163363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.163305    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\") pod \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") "
Oct 30 04:17:37.163363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.163342    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\" (UniqueName: \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\") pod \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") "
Oct 30 04:17:37.163504 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.163385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\") pod \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") "
Oct 30 04:17:37.163504 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.163418    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\") pod \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") "
Oct 30 04:17:37.163840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.163802    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal" pod "30108c6c-1ed0-45a7-9974-24f0e7f03c0b" (UID: "30108c6c-1ed0-45a7-9974-24f0e7f03c0b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:37.165330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.165310    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs" pod "30108c6c-1ed0-45a7-9974-24f0e7f03c0b" (UID: "30108c6c-1ed0-45a7-9974-24f0e7f03c0b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:37.165394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.165343    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs" pod "30108c6c-1ed0-45a7-9974-24f0e7f03c0b" (UID: "30108c6c-1ed0-45a7-9974-24f0e7f03c0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:37.165465 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.165448    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw" pod "30108c6c-1ed0-45a7-9974-24f0e7f03c0b" (UID: "30108c6c-1ed0-45a7-9974-24f0e7f03c0b"). InnerVolumeSpecName "kube-api-access-vzvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:37.263912 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.263878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\") pod \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\" (UID: \"30108c6c-1ed0-45a7-9974-24f0e7f03c0b\") "
Oct 30 04:17:37.264079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.264062    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:37.264139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.264087    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvcw\" (UniqueName: \"kubernetes.io/projected/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-kube-api-access-vzvcw\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:37.264139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.264102    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:37.264139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.264116    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:37.265875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.265855    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert" pod "30108c6c-1ed0-45a7-9974-24f0e7f03c0b" (UID: "30108c6c-1ed0-45a7-9974-24f0e7f03c0b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:17:37.364510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.364480    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/30108c6c-1ed0-45a7-9974-24f0e7f03c0b-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:17:37.516903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.516796    4528 generic.go:356] "Generic (PLEG): container finished" podID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" containerID="374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915" exitCode=0
Oct 30 04:17:37.516903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.516860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.516903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.516868    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30108c6c-1ed0-45a7-9974-24f0e7f03c0b","Type":"ContainerDied","Data":"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"}
Oct 30 04:17:37.517202 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.516935    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"30108c6c-1ed0-45a7-9974-24f0e7f03c0b","Type":"ContainerDied","Data":"44cca91dc9b08e16659da1fd931ae8349b18731d1ff1d15907c9f5107635d800"}
Oct 30 04:17:37.517202 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.516956    4528 scope.go:122] "RemoveContainer" containerID="374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"
Oct 30 04:17:37.525900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.525881    4528 scope.go:122] "RemoveContainer" containerID="374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"
Oct 30 04:17:37.526173 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:17:37.526151    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915\": container with ID starting with 374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915 not found: ID does not exist" containerID="374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"
Oct 30 04:17:37.526271 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.526180    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915"} err="failed to get container status \"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915\": rpc error: code = NotFound desc = could not find container \"374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915\": container with ID starting with 374257f1b0721ac91cebc78c22f53e831fd7fad8898bede212be7a20a02cf915 not found: ID does not exist"
Oct 30 04:17:37.566327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.566293    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:37.577962 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.577934    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:37.607410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.607383    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:37.607694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.607681    4528 cpu_manager.go:414] "containerMap: removing container" podUID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" containerName="otc-container"
Oct 30 04:17:37.607737 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.607696    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" containerName="otc-container"
Oct 30 04:17:37.607775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.607767    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" containerName="otc-container"
Oct 30 04:17:37.611921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.611905    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.626501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.626475    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:17:37.626648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.626523    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:17:37.626648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.626539    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:17:37.626648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.626528    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\""
Oct 30 04:17:37.626648 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.626592    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:17:37.632957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.632926    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:37.667277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.667246    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.667443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.667286    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.667443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.667411    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.667528 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.667450    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.667528 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.667472    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9tn\" (UniqueName: \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768053    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768126 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768093    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9tn\" (UniqueName: \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768131    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768151    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.768898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.768873    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.770572 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.770538    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.770674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.770575    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.770674 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.770653    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.791111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.791091    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9tn\" (UniqueName: \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\") pod \"data-science-collector-collector-0\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:37.893217 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.893180    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30108c6c-1ed0-45a7-9974-24f0e7f03c0b" path="/var/lib/kubelet/pods/30108c6c-1ed0-45a7-9974-24f0e7f03c0b/volumes"
Oct 30 04:17:37.923993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:37.923966    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:17:38.221658 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.221625    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:17:38.224691 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:17:38.224665    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf752b6_00cd_4e26_bf28_9ddeb41ccf7e.slice/crio-2c837ec68b8500a462bdf303388eeb8da1b0cb82d910559472e78d2f3b64a072 WatchSource:0}: Error finding container 2c837ec68b8500a462bdf303388eeb8da1b0cb82d910559472e78d2f3b64a072: Status 404 returned error can't find the container with id 2c837ec68b8500a462bdf303388eeb8da1b0cb82d910559472e78d2f3b64a072
Oct 30 04:17:38.226870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.226814    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:17:38.226988 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.226908    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:17:38.524102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.524023    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e","Type":"ContainerStarted","Data":"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"}
Oct 30 04:17:38.524102 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.524076    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e","Type":"ContainerStarted","Data":"2c837ec68b8500a462bdf303388eeb8da1b0cb82d910559472e78d2f3b64a072"}
Oct 30 04:17:38.565209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:38.565156    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.565143199 podStartE2EDuration="1.565143199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:38.563911034 +0000 UTC m=+5816.788160311" watchObservedRunningTime="2026-10-30 04:17:38.565143199 +0000 UTC m=+5816.789392473"
Oct 30 04:17:41.675593 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:41.675555    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:58.908853 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:58.908776    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:17:59.880101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:17:59.880065    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:00.001231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:00.001197    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:00.122719 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:00.122673    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:01.750017 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:01.749972    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:01.882015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:01.881982    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:02.002919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:02.002793    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:04.057455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:04.057422    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:04.058207 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:04.058170    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" containerName="otc-container" containerID="cri-o://948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba" gracePeriod=30
Oct 30 04:18:04.633130 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:04.633096    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:18:06.407135 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.407114    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.498204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498118    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\") pod \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") "
Oct 30 04:18:06.498204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498183    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\") pod \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") "
Oct 30 04:18:06.498447 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498213    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\") pod \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") "
Oct 30 04:18:06.498447 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498240    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\") pod \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") "
Oct 30 04:18:06.498447 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498289    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\" (UniqueName: \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\") pod \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\" (UID: \"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e\") "
Oct 30 04:18:06.498662 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.498637    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal" pod "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" (UID: "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:06.500483 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.500456    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert" pod "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" (UID: "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:06.500587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.500535    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn" pod "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" (UID: "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e"). InnerVolumeSpecName "kube-api-access-qp9tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:06.500753 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.500735    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs" pod "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" (UID: "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:06.500811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.500757    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs" pod "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" (UID: "eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:06.553523 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.553490    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:18:06.553682 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.553661    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" podUID="00d7b8f1-526f-46ab-91ab-5b19904635c4" containerName="thanos-querier" containerID="cri-o://c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c" gracePeriod=30
Oct 30 04:18:06.591640 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.591602    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"]
Oct 30 04:18:06.591964 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.591946    4528 cpu_manager.go:414] "containerMap: removing container" podUID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" containerName="otc-container"
Oct 30 04:18:06.591964 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.591963    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" containerName="otc-container"
Oct 30 04:18:06.592116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.592039    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" containerName="otc-container"
Oct 30 04:18:06.595289 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.595260    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"
Oct 30 04:18:06.599174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.599139    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:06.599318 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.599177    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:06.599318 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.599193    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:06.599318 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.599205    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:06.599318 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.599218    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9tn\" (UniqueName: \"kubernetes.io/projected/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e-kube-api-access-qp9tn\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:06.607665 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.607632    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"]
Oct 30 04:18:06.624487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.624460    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:06.627875 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:06.627849    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hnjwk], unattached volumes=[], failed to process volumes=[kube-api-access-hnjwk]: context canceled" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j" podUID="5425020e-ae14-495a-8ce5-3470069be851"
Oct 30 04:18:06.650214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.650176    4528 generic.go:356] "Generic (PLEG): container finished" podID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" containerID="948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba" exitCode=0
Oct 30 04:18:06.650400 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.650336    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"
Oct 30 04:18:06.650796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.650774    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.652322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.651922    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e","Type":"ContainerDied","Data":"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"}
Oct 30 04:18:06.652322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.652044    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e","Type":"ContainerDied","Data":"2c837ec68b8500a462bdf303388eeb8da1b0cb82d910559472e78d2f3b64a072"}
Oct 30 04:18:06.652322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.652088    4528 scope.go:122] "RemoveContainer" containerID="948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"
Oct 30 04:18:06.661107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.660998    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"
Oct 30 04:18:06.671033 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.670112    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"]
Oct 30 04:18:06.671033 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.670910    4528 scope.go:122] "RemoveContainer" containerID="948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"
Oct 30 04:18:06.671677 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:06.671278    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba\": container with ID starting with 948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba not found: ID does not exist" containerID="948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"
Oct 30 04:18:06.671677 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.671313    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba"} err="failed to get container status \"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba\": rpc error: code = NotFound desc = could not find container \"948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba\": container with ID starting with 948203824d8d740257fc4c3b479e8dcba394c850a6ad4f59c5b26dde868b6aba not found: ID does not exist"
Oct 30 04:18:06.725246 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.725207    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:06.725496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.725470    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus" containerID="cri-o://842c58c64aac0ae8b1fe3a540b2b2c8deb0f99f3137e25660bab6e58c8d85e95" gracePeriod=600
Oct 30 04:18:06.725650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.725629    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="thanos-sidecar" containerID="cri-o://29632cd4c11e675150ff47d346f643390a8677c3b9b9eb7782af8cd9387701cc" gracePeriod=600
Oct 30 04:18:06.725745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.725695    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="config-reloader" containerID="cri-o://6529f5fe18f133004538ba6e9b795105328a4dca6eb2289d3928da949302e61d" gracePeriod=600
Oct 30 04:18:06.753643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.753578    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:06.772658 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.772627    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:06.778552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.778524    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:06.814947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.814913    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:06.818249 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.818233    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.837710 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.837683    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:18:06.837710 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.837702    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-m9xwj\""
Oct 30 04:18:06.837907 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.837733    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:18:06.837907 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.837746    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:18:06.837907 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.837753    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:18:06.842452 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.840410    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:06.878535 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.878498    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:06.901616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.901591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.901738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.901623    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8zh\" (UniqueName: \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.901738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.901711    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.901842 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.901744    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.901896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.901877    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:06.937397 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:06.937372    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:18:07.003263 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003232    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\" (UniqueName: \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\") pod \"00d7b8f1-526f-46ab-91ab-5b19904635c4\" (UID: \"00d7b8f1-526f-46ab-91ab-5b19904635c4\") "
Oct 30 04:18:07.003431 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8zh\" (UniqueName: \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.003431 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003399    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.003521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003455    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.003521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003505    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.003617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.003545    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.004361 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.004288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.005623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.005594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67" pod "00d7b8f1-526f-46ab-91ab-5b19904635c4" (UID: "00d7b8f1-526f-46ab-91ab-5b19904635c4"). InnerVolumeSpecName "kube-api-access-phx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:07.005910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.005894    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.005966 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.005931    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.006035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.006017    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.025056 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.025026    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:07.026271 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.026253    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8zh\" (UniqueName: \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\") pod \"data-science-collector-collector-0\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.104812 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.104778    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phx67\" (UniqueName: \"kubernetes.io/projected/00d7b8f1-526f-46ab-91ab-5b19904635c4-kube-api-access-phx67\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:07.131225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.131196    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:07.132909 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.132883    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:07.351397 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.351308    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:07.386746 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.386716    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:07.428989 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.428596    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:07.431715 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:18:07.431680    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e0073_24a7_4854_951c_3d7015cc6d5f.slice/crio-3093d7cce3a8fe521249f9eff0d9856d67d009be0d3eeeb87db8467f7dd0bf55 WatchSource:0}: Error finding container 3093d7cce3a8fe521249f9eff0d9856d67d009be0d3eeeb87db8467f7dd0bf55: Status 404 returned error can't find the container with id 3093d7cce3a8fe521249f9eff0d9856d67d009be0d3eeeb87db8467f7dd0bf55
Oct 30 04:18:07.434472 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.434442    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:07.434579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.434500    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:07.505797 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.505765    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:07.621299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.621221    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:07.656107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.656079    4528 generic.go:356] "Generic (PLEG): container finished" podID="00d7b8f1-526f-46ab-91ab-5b19904635c4" containerID="c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c" exitCode=0
Oct 30 04:18:07.656267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.656124    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"
Oct 30 04:18:07.656267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.656161    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" event={"ID":"00d7b8f1-526f-46ab-91ab-5b19904635c4","Type":"ContainerDied","Data":"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"}
Oct 30 04:18:07.656267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.656204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29" event={"ID":"00d7b8f1-526f-46ab-91ab-5b19904635c4","Type":"ContainerDied","Data":"3172b585845a30f64b3232a0929458376da90309483a4f23557a55c2eeb26cfc"}
Oct 30 04:18:07.656267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.656224    4528 scope.go:122] "RemoveContainer" containerID="c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"
Oct 30 04:18:07.659308 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.659283    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c10e0073-24a7-4854-951c-3d7015cc6d5f","Type":"ContainerStarted","Data":"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"}
Oct 30 04:18:07.659432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.659332    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c10e0073-24a7-4854-951c-3d7015cc6d5f","Type":"ContainerStarted","Data":"3093d7cce3a8fe521249f9eff0d9856d67d009be0d3eeeb87db8467f7dd0bf55"}
Oct 30 04:18:07.659432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.659335    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="c10e0073-24a7-4854-951c-3d7015cc6d5f" containerName="otc-container" containerID="cri-o://e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964" gracePeriod=30
Oct 30 04:18:07.662236 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.662211    4528 generic.go:356] "Generic (PLEG): container finished" podID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerID="29632cd4c11e675150ff47d346f643390a8677c3b9b9eb7782af8cd9387701cc" exitCode=0
Oct 30 04:18:07.662356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.662237    4528 generic.go:356] "Generic (PLEG): container finished" podID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerID="6529f5fe18f133004538ba6e9b795105328a4dca6eb2289d3928da949302e61d" exitCode=0
Oct 30 04:18:07.662356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.662271    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerDied","Data":"29632cd4c11e675150ff47d346f643390a8677c3b9b9eb7782af8cd9387701cc"}
Oct 30 04:18:07.662356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.662313    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"
Oct 30 04:18:07.662356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.662327    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerDied","Data":"6529f5fe18f133004538ba6e9b795105328a4dca6eb2289d3928da949302e61d"}
Oct 30 04:18:07.669531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.669510    4528 scope.go:122] "RemoveContainer" containerID="c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"
Oct 30 04:18:07.670146 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:07.670116    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c\": container with ID starting with c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c not found: ID does not exist" containerID="c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"
Oct 30 04:18:07.670254 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.670157    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c"} err="failed to get container status \"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c\": rpc error: code = NotFound desc = could not find container \"c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c\": container with ID starting with c7f86c77ae045112e1b06768d417739cc241a275018a49a5566cd81177f0433c not found: ID does not exist"
Oct 30 04:18:07.703451 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.703417    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:18:07.707108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.707078    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-tpk29"]
Oct 30 04:18:07.746356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.746303    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.746286698 podStartE2EDuration="1.746286698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:07.746269539 +0000 UTC m=+5845.970518817" watchObservedRunningTime="2026-10-30 04:18:07.746286698 +0000 UTC m=+5845.970536035"
Oct 30 04:18:07.816283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.816244    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"]
Oct 30 04:18:07.819890 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.819867    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8wk5j"]
Oct 30 04:18:07.892118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.892093    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d7b8f1-526f-46ab-91ab-5b19904635c4" path="/var/lib/kubelet/pods/00d7b8f1-526f-46ab-91ab-5b19904635c4/volumes"
Oct 30 04:18:07.892402 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.892388    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5425020e-ae14-495a-8ce5-3470069be851" path="/var/lib/kubelet/pods/5425020e-ae14-495a-8ce5-3470069be851/volumes"
Oct 30 04:18:07.892663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.892646    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e" path="/var/lib/kubelet/pods/eaf752b6-00cd-4e26-bf28-9ddeb41ccf7e/volumes"
Oct 30 04:18:07.896253 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:07.896234    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:08.010884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.010853    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\") pod \"c10e0073-24a7-4854-951c-3d7015cc6d5f\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") "
Oct 30 04:18:08.010884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.010887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\") pod \"c10e0073-24a7-4854-951c-3d7015cc6d5f\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") "
Oct 30 04:18:08.011081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.010913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\") pod \"c10e0073-24a7-4854-951c-3d7015cc6d5f\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") "
Oct 30 04:18:08.011081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.011005    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\" (UniqueName: \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\") pod \"c10e0073-24a7-4854-951c-3d7015cc6d5f\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") "
Oct 30 04:18:08.011081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.011044    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\") pod \"c10e0073-24a7-4854-951c-3d7015cc6d5f\" (UID: \"c10e0073-24a7-4854-951c-3d7015cc6d5f\") "
Oct 30 04:18:08.011270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.011246    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal" pod "c10e0073-24a7-4854-951c-3d7015cc6d5f" (UID: "c10e0073-24a7-4854-951c-3d7015cc6d5f"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:08.013031 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.013001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs" pod "c10e0073-24a7-4854-951c-3d7015cc6d5f" (UID: "c10e0073-24a7-4854-951c-3d7015cc6d5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:08.013166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.013081    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert" pod "c10e0073-24a7-4854-951c-3d7015cc6d5f" (UID: "c10e0073-24a7-4854-951c-3d7015cc6d5f"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:08.013166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.013102    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs" pod "c10e0073-24a7-4854-951c-3d7015cc6d5f" (UID: "c10e0073-24a7-4854-951c-3d7015cc6d5f"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:08.013166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.013101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh" pod "c10e0073-24a7-4854-951c-3d7015cc6d5f" (UID: "c10e0073-24a7-4854-951c-3d7015cc6d5f"). InnerVolumeSpecName "kube-api-access-lb8zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:08.111911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.111872    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8zh\" (UniqueName: \"kubernetes.io/projected/c10e0073-24a7-4854-951c-3d7015cc6d5f-kube-api-access-lb8zh\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:08.111911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.111906    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:08.112114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.111920    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:08.112114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.111932    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c10e0073-24a7-4854-951c-3d7015cc6d5f-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:08.112114 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.111943    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e0073-24a7-4854-951c-3d7015cc6d5f-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:08.367130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.367096    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:08.441212 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.441172    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:18:08.484834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.484721    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:08.604576 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.604539    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:08.669932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.669900    4528 generic.go:356] "Generic (PLEG): container finished" podID="c10e0073-24a7-4854-951c-3d7015cc6d5f" containerID="e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964" exitCode=2
Oct 30 04:18:08.670080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.669944    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:08.670080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.669979    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c10e0073-24a7-4854-951c-3d7015cc6d5f","Type":"ContainerDied","Data":"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"}
Oct 30 04:18:08.670080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.670021    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"c10e0073-24a7-4854-951c-3d7015cc6d5f","Type":"ContainerDied","Data":"3093d7cce3a8fe521249f9eff0d9856d67d009be0d3eeeb87db8467f7dd0bf55"}
Oct 30 04:18:08.670080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.670046    4528 scope.go:122] "RemoveContainer" containerID="e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"
Oct 30 04:18:08.678728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.678710    4528 scope.go:122] "RemoveContainer" containerID="e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"
Oct 30 04:18:08.679051 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:08.679029    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964\": container with ID starting with e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964 not found: ID does not exist" containerID="e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"
Oct 30 04:18:08.679150 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.679065    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964"} err="failed to get container status \"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964\": rpc error: code = NotFound desc = could not find container \"e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964\": container with ID starting with e9cfbe91380d061ab116ca0328193aee36410ae20bb19a98d46f24ad07f88964 not found: ID does not exist"
Oct 30 04:18:08.720327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.720295    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:08.722468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:08.722443    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:09.892371 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:09.892342    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c10e0073-24a7-4854-951c-3d7015cc6d5f" path="/var/lib/kubelet/pods/c10e0073-24a7-4854-951c-3d7015cc6d5f/volumes"
Oct 30 04:18:13.439609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:13.439566    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:18:18.439488 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:18.439444    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:18:18.439994 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:18.439607    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:23.440310 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.440272    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:18:23.734626 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.734595    4528 generic.go:356] "Generic (PLEG): container finished" podID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerID="842c58c64aac0ae8b1fe3a540b2b2c8deb0f99f3137e25660bab6e58c8d85e95" exitCode=0
Oct 30 04:18:23.734784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.734663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerDied","Data":"842c58c64aac0ae8b1fe3a540b2b2c8deb0f99f3137e25660bab6e58c8d85e95"}
Oct 30 04:18:23.773131 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.773106    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:23.848589 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.848800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848596    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.848800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848631    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.848800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848735    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.848800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848781    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848850    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848931    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848967    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.848997    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849032    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849064    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849106    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\" (UID: \"165df98a-1ed0-4ede-b5a8-a45ea59f493c\") "
Oct 30 04:18:23.849473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849291    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:23.849473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849368    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.850027 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.849980    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:23.850377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.850345    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:23.851634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.851543    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "kube-api-access-zlps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:23.851926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.851889    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.852235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.852197    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.853093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.853023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:23.853093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.853035    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:23.853258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.853087    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.853479 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.853456    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.853479 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.853469    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.855055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.855036    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:18:23.864526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.864498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config" pod "165df98a-1ed0-4ede-b5a8-a45ea59f493c" (UID: "165df98a-1ed0-4ede-b5a8-a45ea59f493c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950488    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950512    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950524    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950535    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950546    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950554    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950558 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950564    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950573    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlps8\" (UniqueName: \"kubernetes.io/projected/165df98a-1ed0-4ede-b5a8-a45ea59f493c-kube-api-access-zlps8\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950581    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/165df98a-1ed0-4ede-b5a8-a45ea59f493c-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950606    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:18:23.950906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950614    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/165df98a-1ed0-4ede-b5a8-a45ea59f493c-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.950906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.950623    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/165df98a-1ed0-4ede-b5a8-a45ea59f493c-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:23.961286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:23.961268    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:18:24.051145 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.051104    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:18:24.740850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.740800    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"165df98a-1ed0-4ede-b5a8-a45ea59f493c","Type":"ContainerDied","Data":"f2711e232c230f361d8af95d933a0339e8e351301a1ccc712cb54e4c33141ab5"}
Oct 30 04:18:24.740850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.740833    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:24.741335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.740886    4528 scope.go:122] "RemoveContainer" containerID="29632cd4c11e675150ff47d346f643390a8677c3b9b9eb7782af8cd9387701cc"
Oct 30 04:18:24.748885 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.748863    4528 scope.go:122] "RemoveContainer" containerID="6529f5fe18f133004538ba6e9b795105328a4dca6eb2289d3928da949302e61d"
Oct 30 04:18:24.756584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.756568    4528 scope.go:122] "RemoveContainer" containerID="842c58c64aac0ae8b1fe3a540b2b2c8deb0f99f3137e25660bab6e58c8d85e95"
Oct 30 04:18:24.763979 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.763954    4528 scope.go:122] "RemoveContainer" containerID="9f7c0e0a68466febebc3cd9aab23f7a74afcda75e8f40664fde1e7c00503a783"
Oct 30 04:18:24.804984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.804945    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:24.816512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:24.816480    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:25.891636 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:25.891604    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" path="/var/lib/kubelet/pods/165df98a-1ed0-4ede-b5a8-a45ea59f493c/volumes"
Oct 30 04:18:27.482725 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:27.482676    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:27.603666 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:27.603624    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:27.716784 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:27.716751    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:27.841834 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:27.841743    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:28.182105 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:28.182066    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:28.442943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:28.442868    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:28.585116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:28.585076    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:28.778688 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:28.778605    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:29.355587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355555    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:29.355852 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355837    4528 cpu_manager.go:414] "containerMap: removing container" podUID="00d7b8f1-526f-46ab-91ab-5b19904635c4" containerName="thanos-querier"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355853    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d7b8f1-526f-46ab-91ab-5b19904635c4" containerName="thanos-querier"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355862    4528 cpu_manager.go:414] "containerMap: removing container" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="init-config-reloader"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355869    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="init-config-reloader"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355876    4528 cpu_manager.go:414] "containerMap: removing container" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="thanos-sidecar"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355882    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="thanos-sidecar"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355900    4528 cpu_manager.go:414] "containerMap: removing container" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus"
Oct 30 04:18:29.355905 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355905    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355912    4528 cpu_manager.go:414] "containerMap: removing container" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="config-reloader"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355916    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="config-reloader"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355925    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c10e0073-24a7-4854-951c-3d7015cc6d5f" containerName="otc-container"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355930    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c10e0073-24a7-4854-951c-3d7015cc6d5f" containerName="otc-container"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355977    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d7b8f1-526f-46ab-91ab-5b19904635c4" containerName="thanos-querier"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355984    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="thanos-sidecar"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355994    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="prometheus"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.355999    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="165df98a-1ed0-4ede-b5a8-a45ea59f493c" containerName="config-reloader"
Oct 30 04:18:29.356117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.356004    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c10e0073-24a7-4854-951c-3d7015cc6d5f" containerName="otc-container"
Oct 30 04:18:29.360743 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.360720    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.375106 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.375082    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:18:29.386991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.386964    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:18:29.387186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.387004    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\""
Oct 30 04:18:29.387186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.387070    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:18:29.387320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.387273    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:18:29.388407 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.388385    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:29.393866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.393839    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:29.409287 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.409255    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:18:29.475767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.475736    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:18:29.493073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.493042    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.493231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.493080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4j9\" (UniqueName: \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.493231 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.493110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.493307 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.493245    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.493307 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.493286    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.528023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.527987    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:29.594113 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.594552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594125    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.594552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.594552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594205    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4j9\" (UniqueName: \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.594552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.595023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.594998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.596696 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.596677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.596696 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.596685    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.596861 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.596678    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.616344 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.616295    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4j9\" (UniqueName: \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\") pod \"data-science-collector-collector-0\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.649914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.649884    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:29.675403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.675358    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:29.972590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.972561    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:29.975530 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:18:29.975499    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa04fec_180c_4a2d_8ed4_9f655a4e2887.slice/crio-d74f83a80432fc1f255167133059f672907cc24ec3e55ac55fd41cb4b57e0a54 WatchSource:0}: Error finding container d74f83a80432fc1f255167133059f672907cc24ec3e55ac55fd41cb4b57e0a54: Status 404 returned error can't find the container with id d74f83a80432fc1f255167133059f672907cc24ec3e55ac55fd41cb4b57e0a54
Oct 30 04:18:29.977526 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.977493    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:29.977595 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:29.977550    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:30.766594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:30.766558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ffa04fec-180c-4a2d-8ed4-9f655a4e2887","Type":"ContainerStarted","Data":"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"}
Oct 30 04:18:30.767046 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:30.766613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ffa04fec-180c-4a2d-8ed4-9f655a4e2887","Type":"ContainerStarted","Data":"d74f83a80432fc1f255167133059f672907cc24ec3e55ac55fd41cb4b57e0a54"}
Oct 30 04:18:30.807011 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:30.806952    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.806934651 podStartE2EDuration="1.806934651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:30.806448943 +0000 UTC m=+5869.030698220" watchObservedRunningTime="2026-10-30 04:18:30.806934651 +0000 UTC m=+5869.031183929"
Oct 30 04:18:32.103945 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.103909    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:18:32.106695 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.106678    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:18:32.120921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.120900    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:18:32.133321 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.133295    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:18:32.215717 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.215683    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg7x\" (UniqueName: \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s\" (UID: \"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:18:32.316619 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.316579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg7x\" (UniqueName: \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s\" (UID: \"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:18:32.321242 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.321215    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:32.339468 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.339436    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg7x\" (UniqueName: \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s\" (UID: \"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:18:32.417798 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.417767    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:18:32.564339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.564303    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:32.677327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.677251    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:32.716458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.716426    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:18:32.719500 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:18:32.719472    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa45f3fb_9b72_4a7a_a150_c9263ff6c7af.slice/crio-56a15474d991b38a07d2001ee856285b2749a82da2a3f5b17b92efa5c92a1deb WatchSource:0}: Error finding container 56a15474d991b38a07d2001ee856285b2749a82da2a3f5b17b92efa5c92a1deb: Status 404 returned error can't find the container with id 56a15474d991b38a07d2001ee856285b2749a82da2a3f5b17b92efa5c92a1deb
Oct 30 04:18:32.775830 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.775787    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" event={"ID":"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af","Type":"ContainerStarted","Data":"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"}
Oct 30 04:18:32.775958 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.775857    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" event={"ID":"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af","Type":"ContainerStarted","Data":"56a15474d991b38a07d2001ee856285b2749a82da2a3f5b17b92efa5c92a1deb"}
Oct 30 04:18:32.813793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:32.813740    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" podStartSLOduration=0.813723664 podStartE2EDuration="813.723664ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:32.813069651 +0000 UTC m=+5871.037318928" watchObservedRunningTime="2026-10-30 04:18:32.813723664 +0000 UTC m=+5871.037972941"
Oct 30 04:18:33.094591 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.094503    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:33.097247 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.097218    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:33.098195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.098179    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.113557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.113535    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:18:33.113973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.113701    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:18:33.113973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.113754    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:18:33.128342 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.128310    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:33.208507 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208471    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:18:33.208676 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208534    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:18:33.208780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208749    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:18:33.208780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208765    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:18:33.208780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208754    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:18:33.209015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208845    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:18:33.209015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.208871    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-7vtv5\""
Oct 30 04:18:33.220524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.220498    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:33.223538 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223518    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223624 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223546    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223624 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223624 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223591    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223771 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223668    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223851 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223800    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223938 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223878    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223939    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.223981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223962    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvz9n\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.224091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.223981    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.224091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.224006    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.224091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.224044    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.324862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324812    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.324862 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324862    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324888    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324925    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324946    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.324972    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325002    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325028    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvz9n\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325082    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325121 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325105    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325619    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.325672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.325622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.326506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.326478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328353    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328548    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.328654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.328629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.329041 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.329023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:33.348951 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:33.348886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvz9n\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.332673 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.332629    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.372216 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.372181    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.433504 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.433471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.433694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.433592    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.447355 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.447327    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.545811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.545781    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.571378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.571350    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.617788 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.617703    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:34.631433 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:34.631391    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:18:34.957460 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:18:34.957427    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1370229c_ea5d_4fcf_8b2c_55196fa676f2.slice/crio-c6bdcf0a8f64dbe11cc87faae19c27d1f30665fa2336e964f276f09917afb7ee WatchSource:0}: Error finding container c6bdcf0a8f64dbe11cc87faae19c27d1f30665fa2336e964f276f09917afb7ee: Status 404 returned error can't find the container with id c6bdcf0a8f64dbe11cc87faae19c27d1f30665fa2336e964f276f09917afb7ee
Oct 30 04:18:34.966770 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:34.966746    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:18:35.790006 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:35.789968    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerStarted","Data":"cb004a8051f03f65add505f6654f0d955c7daf8b76c8fd581127e53b85bd9bb2"}
Oct 30 04:18:35.790394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:35.790033    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerStarted","Data":"c6bdcf0a8f64dbe11cc87faae19c27d1f30665fa2336e964f276f09917afb7ee"}
Oct 30 04:18:38.803623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:38.803591    4528 generic.go:356] "Generic (PLEG): container finished" podID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerID="cb004a8051f03f65add505f6654f0d955c7daf8b76c8fd581127e53b85bd9bb2" exitCode=0
Oct 30 04:18:38.803990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:38.803665    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerDied","Data":"cb004a8051f03f65add505f6654f0d955c7daf8b76c8fd581127e53b85bd9bb2"}
Oct 30 04:18:39.810133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:39.810095    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerStarted","Data":"16f1ea4dbda5a78cabc9dd5e0f12998c054e75bde87aa42f7ceb2dde5011f7c6"}
Oct 30 04:18:39.810605 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:39.810175    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerStarted","Data":"d8ed11451a89cd49ea7d668f9d54249605565288576b7cf785b63733dd036e15"}
Oct 30 04:18:39.810605 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:39.810214    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerStarted","Data":"e64057faceec5841c38467923a122fd150c0df758b85da1c7257032d9ad595b5"}
Oct 30 04:18:39.894855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:39.894784    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=3.509502324 podStartE2EDuration="6.894771548s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.385269224s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:39.882854236 +0000 UTC m=+5878.107103513" watchObservedRunningTime="2026-10-30 04:18:39.894771548 +0000 UTC m=+5878.119020825"
Oct 30 04:18:41.773417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:41.773376    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:44.617919 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:44.617879    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:48.762249 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:48.762215    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:48.889805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:48.889769    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:49.091084 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:49.091002    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:49.618670 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:49.618637    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:49.624146 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:49.624122    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:49.858870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:49.858844    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:18:50.645500 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:50.645463    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:50.766460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:50.766424    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:50.968129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:50.968036    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:18:53.332929 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.332891    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:53.333699 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.333640    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" containerName="otc-container" containerID="cri-o://cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f" gracePeriod=30
Oct 30 04:18:53.577383 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.577356    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:53.706011 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.705979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\" (UniqueName: \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\") pod \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") "
Oct 30 04:18:53.706223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.706026    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\") pod \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") "
Oct 30 04:18:53.706223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.706045    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\") pod \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") "
Oct 30 04:18:53.706223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.706066    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\") pod \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") "
Oct 30 04:18:53.706223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.706095    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\") pod \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\" (UID: \"ffa04fec-180c-4a2d-8ed4-9f655a4e2887\") "
Oct 30 04:18:53.706474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.706426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal" pod "ffa04fec-180c-4a2d-8ed4-9f655a4e2887" (UID: "ffa04fec-180c-4a2d-8ed4-9f655a4e2887"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:18:53.708521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.708488    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs" pod "ffa04fec-180c-4a2d-8ed4-9f655a4e2887" (UID: "ffa04fec-180c-4a2d-8ed4-9f655a4e2887"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:53.708654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.708516    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs" pod "ffa04fec-180c-4a2d-8ed4-9f655a4e2887" (UID: "ffa04fec-180c-4a2d-8ed4-9f655a4e2887"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:53.708756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.708735    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert" pod "ffa04fec-180c-4a2d-8ed4-9f655a4e2887" (UID: "ffa04fec-180c-4a2d-8ed4-9f655a4e2887"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:53.708839 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.708770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9" pod "ffa04fec-180c-4a2d-8ed4-9f655a4e2887" (UID: "ffa04fec-180c-4a2d-8ed4-9f655a4e2887"). InnerVolumeSpecName "kube-api-access-2k4j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:53.807497 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.807462    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4j9\" (UniqueName: \"kubernetes.io/projected/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-kube-api-access-2k4j9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.807497 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.807490    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.807497 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.807500    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.807741 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.807510    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.807741 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.807519    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ffa04fec-180c-4a2d-8ed4-9f655a4e2887-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:18:53.871019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.870987    4528 generic.go:356] "Generic (PLEG): container finished" podID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" containerID="cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f" exitCode=0
Oct 30 04:18:53.871162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.871032    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ffa04fec-180c-4a2d-8ed4-9f655a4e2887","Type":"ContainerDied","Data":"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"}
Oct 30 04:18:53.871162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.871044    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:53.871162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.871074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ffa04fec-180c-4a2d-8ed4-9f655a4e2887","Type":"ContainerDied","Data":"d74f83a80432fc1f255167133059f672907cc24ec3e55ac55fd41cb4b57e0a54"}
Oct 30 04:18:53.871162 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.871098    4528 scope.go:122] "RemoveContainer" containerID="cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"
Oct 30 04:18:53.879850 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.879809    4528 scope.go:122] "RemoveContainer" containerID="cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"
Oct 30 04:18:53.880111 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:18:53.880094    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f\": container with ID starting with cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f not found: ID does not exist" containerID="cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"
Oct 30 04:18:53.880157 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.880121    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f"} err="failed to get container status \"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f\": rpc error: code = NotFound desc = could not find container \"cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f\": container with ID starting with cc596fd2d6ad5bb84f4e28baa0f6f07f36bea4773607ccbb487ceb0414ad8d0f not found: ID does not exist"
Oct 30 04:18:53.924618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.924587    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:53.936089 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.936063    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:53.967372 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.967293    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:53.967605 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.967592    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" containerName="otc-container"
Oct 30 04:18:53.967650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.967607    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" containerName="otc-container"
Oct 30 04:18:53.967689 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.967677    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" containerName="otc-container"
Oct 30 04:18:53.971911 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.971893    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:53.986349 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.986325    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:18:53.986462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.986357    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:18:53.986462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.986421    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\""
Oct 30 04:18:53.986462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.986433    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:18:53.986462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.986449    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:18:53.991731 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:53.991706    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:54.110434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.110401    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.110434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.110437    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.110637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.110555    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.110637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.110609    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzhj\" (UniqueName: \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.110637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.110634    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.211631 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.211597    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.211840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.211646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzhj\" (UniqueName: \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.211840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.211681    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.211840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.211723    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.211840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.211751    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.212513 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.212480    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.214218 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.214198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.214301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.214264    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.214301 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.214287    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.234805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.234751    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzhj\" (UniqueName: \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\") pod \"data-science-collector-collector-0\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.283787 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.283752    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:18:54.581881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.581488    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:18:54.584240 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:18:54.584210    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1933c895_779b_44e5_be1d_dfa0004256bc.slice/crio-3257fcaa617f2ab6ea9192f7666b3edd508fe3bb6bb216b0efeb812e26b4ac90 WatchSource:0}: Error finding container 3257fcaa617f2ab6ea9192f7666b3edd508fe3bb6bb216b0efeb812e26b4ac90: Status 404 returned error can't find the container with id 3257fcaa617f2ab6ea9192f7666b3edd508fe3bb6bb216b0efeb812e26b4ac90
Oct 30 04:18:54.586494 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.586466    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:54.586562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.586525    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:18:54.877359 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.877261    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1933c895-779b-44e5-be1d-dfa0004256bc","Type":"ContainerStarted","Data":"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"}
Oct 30 04:18:54.877359 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.877327    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1933c895-779b-44e5-be1d-dfa0004256bc","Type":"ContainerStarted","Data":"3257fcaa617f2ab6ea9192f7666b3edd508fe3bb6bb216b0efeb812e26b4ac90"}
Oct 30 04:18:54.919015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:54.918956    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.918942356 podStartE2EDuration="1.918942356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:54.917716994 +0000 UTC m=+5893.141966272" watchObservedRunningTime="2026-10-30 04:18:54.918942356 +0000 UTC m=+5893.143191630"
Oct 30 04:18:55.891665 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:18:55.891634    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa04fec-180c-4a2d-8ed4-9f655a4e2887" path="/var/lib/kubelet/pods/ffa04fec-180c-4a2d-8ed4-9f655a4e2887/volumes"
Oct 30 04:19:04.255836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:04.255776    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:04.368012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:04.367979    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:04.569188 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:04.569088    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:04.631750 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:04.631715    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:19:04.695380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:04.695343    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:06.186237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:06.186176    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:06.314346 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:06.314310    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:06.426510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:06.426463    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:09.414800 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.414764    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:09.415283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.415020    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="1933c895-779b-44e5-be1d-dfa0004256bc" containerName="otc-container" containerID="cri-o://8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360" gracePeriod=30
Oct 30 04:19:09.656154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.656131    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:09.847021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.846930    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\") pod \"1933c895-779b-44e5-be1d-dfa0004256bc\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") "
Oct 30 04:19:09.847021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.846973    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\" (UniqueName: \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\") pod \"1933c895-779b-44e5-be1d-dfa0004256bc\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") "
Oct 30 04:19:09.847278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.847024    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\") pod \"1933c895-779b-44e5-be1d-dfa0004256bc\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") "
Oct 30 04:19:09.847278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.847053    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\") pod \"1933c895-779b-44e5-be1d-dfa0004256bc\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") "
Oct 30 04:19:09.847278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.847101    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\") pod \"1933c895-779b-44e5-be1d-dfa0004256bc\" (UID: \"1933c895-779b-44e5-be1d-dfa0004256bc\") "
Oct 30 04:19:09.847565 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.847535    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal" pod "1933c895-779b-44e5-be1d-dfa0004256bc" (UID: "1933c895-779b-44e5-be1d-dfa0004256bc"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:09.849219 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.849191    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs" pod "1933c895-779b-44e5-be1d-dfa0004256bc" (UID: "1933c895-779b-44e5-be1d-dfa0004256bc"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:09.849333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.849241    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs" pod "1933c895-779b-44e5-be1d-dfa0004256bc" (UID: "1933c895-779b-44e5-be1d-dfa0004256bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:09.849520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.849500    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj" pod "1933c895-779b-44e5-be1d-dfa0004256bc" (UID: "1933c895-779b-44e5-be1d-dfa0004256bc"). InnerVolumeSpecName "kube-api-access-kkzhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:09.849600 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.849510    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert" pod "1933c895-779b-44e5-be1d-dfa0004256bc" (UID: "1933c895-779b-44e5-be1d-dfa0004256bc"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:09.941511 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.941480    4528 generic.go:356] "Generic (PLEG): container finished" podID="1933c895-779b-44e5-be1d-dfa0004256bc" containerID="8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360" exitCode=0
Oct 30 04:19:09.941656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.941515    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1933c895-779b-44e5-be1d-dfa0004256bc","Type":"ContainerDied","Data":"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"}
Oct 30 04:19:09.941656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.941524    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:09.941656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.941551    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"1933c895-779b-44e5-be1d-dfa0004256bc","Type":"ContainerDied","Data":"3257fcaa617f2ab6ea9192f7666b3edd508fe3bb6bb216b0efeb812e26b4ac90"}
Oct 30 04:19:09.941656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.941572    4528 scope.go:122] "RemoveContainer" containerID="8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"
Oct 30 04:19:09.948226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.948206    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:09.948226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.948227    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzhj\" (UniqueName: \"kubernetes.io/projected/1933c895-779b-44e5-be1d-dfa0004256bc-kube-api-access-kkzhj\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:09.948361 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.948236    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:09.948361 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.948252    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1933c895-779b-44e5-be1d-dfa0004256bc-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:09.948361 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.948264    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1933c895-779b-44e5-be1d-dfa0004256bc-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:09.949484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.949469    4528 scope.go:122] "RemoveContainer" containerID="8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"
Oct 30 04:19:09.949763 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:09.949742    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360\": container with ID starting with 8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360 not found: ID does not exist" containerID="8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"
Oct 30 04:19:09.949801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.949772    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360"} err="failed to get container status \"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360\": rpc error: code = NotFound desc = could not find container \"8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360\": container with ID starting with 8168ebb34a673ed1bc067212db07124b1c50bbefc1ed7ab0a28ac6e161522360 not found: ID does not exist"
Oct 30 04:19:09.990659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:09.990630    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:10.002380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.002352    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:10.034704 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.034675    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:10.034995 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.034983    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1933c895-779b-44e5-be1d-dfa0004256bc" containerName="otc-container"
Oct 30 04:19:10.035055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.034997    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1933c895-779b-44e5-be1d-dfa0004256bc" containerName="otc-container"
Oct 30 04:19:10.035091 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.035068    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1933c895-779b-44e5-be1d-dfa0004256bc" containerName="otc-container"
Oct 30 04:19:10.039224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.039203    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.053538 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.053511    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:19:10.053873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.053854    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:19:10.053873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.053862    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:19:10.054010 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.053882    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8q6hm\""
Oct 30 04:19:10.054010 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.053928    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:19:10.063756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.063732    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:10.148973 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.148943    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.149154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.148978    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.149154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.149018    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.149237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.149146    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.149237 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.149185    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtq88\" (UniqueName: \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250161 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250120    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq88\" (UniqueName: \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250312    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.250998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.250971    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.252694 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.252667    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.252773 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.252733    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.252840 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.252771    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.273914 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.273888    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq88\" (UniqueName: \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\") pod \"data-science-collector-collector-0\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.351192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.351145    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:10.652757 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.652662    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:10.657289 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:19:10.657258    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81616dfd_5ea0_4992_b8c1_eac1536e6336.slice/crio-2df9742a790f17e7c08280e29582f2670608aff2df61ef4aa37125fdf296e702 WatchSource:0}: Error finding container 2df9742a790f17e7c08280e29582f2670608aff2df61ef4aa37125fdf296e702: Status 404 returned error can't find the container with id 2df9742a790f17e7c08280e29582f2670608aff2df61ef4aa37125fdf296e702
Oct 30 04:19:10.660453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.660416    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:10.660575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.660491    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:10.670793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.670756    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:19:10.671306 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.671024    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" podUID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" containerName="thanos-querier" containerID="cri-o://7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32" gracePeriod=30
Oct 30 04:19:10.734921 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.734787    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:10.807487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.807453    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:10.808547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.808189    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="prometheus" containerID="cri-o://e64057faceec5841c38467923a122fd150c0df758b85da1c7257032d9ad595b5" gracePeriod=600
Oct 30 04:19:10.808699 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.808564    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="thanos-sidecar" containerID="cri-o://16f1ea4dbda5a78cabc9dd5e0f12998c054e75bde87aa42f7ceb2dde5011f7c6" gracePeriod=600
Oct 30 04:19:10.808699 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.808680    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="config-reloader" containerID="cri-o://d8ed11451a89cd49ea7d668f9d54249605565288576b7cf785b63733dd036e15" gracePeriod=600
Oct 30 04:19:10.860610 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.860569    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:10.947329 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.947302    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:19:10.947497 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.947466    4528 generic.go:356] "Generic (PLEG): container finished" podID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" containerID="7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32" exitCode=0
Oct 30 04:19:10.947555 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.947533    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" event={"ID":"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af","Type":"ContainerDied","Data":"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"}
Oct 30 04:19:10.947601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.947582    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s" event={"ID":"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af","Type":"ContainerDied","Data":"56a15474d991b38a07d2001ee856285b2749a82da2a3f5b17b92efa5c92a1deb"}
Oct 30 04:19:10.947650 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.947604    4528 scope.go:122] "RemoveContainer" containerID="7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"
Oct 30 04:19:10.950935 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.950913    4528 generic.go:356] "Generic (PLEG): container finished" podID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerID="16f1ea4dbda5a78cabc9dd5e0f12998c054e75bde87aa42f7ceb2dde5011f7c6" exitCode=0
Oct 30 04:19:10.950935 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.950934    4528 generic.go:356] "Generic (PLEG): container finished" podID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerID="d8ed11451a89cd49ea7d668f9d54249605565288576b7cf785b63733dd036e15" exitCode=0
Oct 30 04:19:10.951101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.950942    4528 generic.go:356] "Generic (PLEG): container finished" podID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerID="e64057faceec5841c38467923a122fd150c0df758b85da1c7257032d9ad595b5" exitCode=0
Oct 30 04:19:10.951101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.951004    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerDied","Data":"16f1ea4dbda5a78cabc9dd5e0f12998c054e75bde87aa42f7ceb2dde5011f7c6"}
Oct 30 04:19:10.951101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.951056    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerDied","Data":"d8ed11451a89cd49ea7d668f9d54249605565288576b7cf785b63733dd036e15"}
Oct 30 04:19:10.951101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.951081    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerDied","Data":"e64057faceec5841c38467923a122fd150c0df758b85da1c7257032d9ad595b5"}
Oct 30 04:19:10.954043 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.953999    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"81616dfd-5ea0-4992-b8c1-eac1536e6336","Type":"ContainerStarted","Data":"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"}
Oct 30 04:19:10.954170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.954045    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"81616dfd-5ea0-4992-b8c1-eac1536e6336","Type":"ContainerStarted","Data":"2df9742a790f17e7c08280e29582f2670608aff2df61ef4aa37125fdf296e702"}
Oct 30 04:19:10.956892 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.956872    4528 scope.go:122] "RemoveContainer" containerID="7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"
Oct 30 04:19:10.957176 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:10.957147    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32\": container with ID starting with 7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32 not found: ID does not exist" containerID="7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"
Oct 30 04:19:10.957236 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.957185    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32"} err="failed to get container status \"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32\": rpc error: code = NotFound desc = could not find container \"7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32\": container with ID starting with 7638d4a31a04256bb4ca4ee8d4cd1352b3df61af8554dee02f51f3283cbf1e32 not found: ID does not exist"
Oct 30 04:19:10.979875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:10.979838    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:11.029051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.028993    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.028972484 podStartE2EDuration="1.028972484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:11.027899166 +0000 UTC m=+5909.252148443" watchObservedRunningTime="2026-10-30 04:19:11.028972484 +0000 UTC m=+5909.253221762"
Oct 30 04:19:11.056034 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.056004    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\" (UniqueName: \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\") pod \"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af\" (UID: \"aa45f3fb-9b72-4a7a-a150-c9263ff6c7af\") "
Oct 30 04:19:11.058385 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.058355    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x" pod "aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" (UID: "aa45f3fb-9b72-4a7a-a150-c9263ff6c7af"). InnerVolumeSpecName "kube-api-access-2wg7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:11.076711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.076683    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:11.157259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.157229    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wg7x\" (UniqueName: \"kubernetes.io/projected/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af-kube-api-access-2wg7x\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.258151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258063    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.258151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.258151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258124    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.258458 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258156    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.258855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258373    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259001 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259001 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.258991    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259138 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259024    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259138 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259076    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259258 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259127    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259460    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\") pod \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\" (UID: \"1370229c-ea5d-4fcf-8b2c-55196fa676f2\") "
Oct 30 04:19:11.259963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.259692    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:11.260891 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.260303    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:11.261325 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.261288    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:11.261957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.261933    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.262089 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262072    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.262269 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262245    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1370229c-ea5d-4fcf-8b2c-55196fa676f2-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.262269 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262115    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "kube-api-access-wvz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:11.262714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.262714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:19:11.262714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262657    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.262714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.262666    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:11.263322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.263295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.263637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.263600    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.264189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.264161    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.265908 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.265888    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:19:11.273866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.273812    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config" pod "1370229c-ea5d-4fcf-8b2c-55196fa676f2" (UID: "1370229c-ea5d-4fcf-8b2c-55196fa676f2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:11.363192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363147    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:19:11.363192 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363189    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363203    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363214    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363223    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363232    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1370229c-ea5d-4fcf-8b2c-55196fa676f2-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363240    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363249    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvz9n\" (UniqueName: \"kubernetes.io/projected/1370229c-ea5d-4fcf-8b2c-55196fa676f2-kube-api-access-wvz9n\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363257    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.363406 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.363266    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/1370229c-ea5d-4fcf-8b2c-55196fa676f2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:11.374971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.374951    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:19:11.464318 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.464287    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:19:11.892239 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.892204    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1933c895-779b-44e5-be1d-dfa0004256bc" path="/var/lib/kubelet/pods/1933c895-779b-44e5-be1d-dfa0004256bc/volumes"
Oct 30 04:19:11.908201 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.908168    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:11.962142 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.962110    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:11.962547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.962108    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"1370229c-ea5d-4fcf-8b2c-55196fa676f2","Type":"ContainerDied","Data":"c6bdcf0a8f64dbe11cc87faae19c27d1f30665fa2336e964f276f09917afb7ee"}
Oct 30 04:19:11.962547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.962287    4528 scope.go:122] "RemoveContainer" containerID="16f1ea4dbda5a78cabc9dd5e0f12998c054e75bde87aa42f7ceb2dde5011f7c6"
Oct 30 04:19:11.963599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.963188    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"
Oct 30 04:19:11.964129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.963638    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-8q6hm\" not found"
Oct 30 04:19:11.972616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.972591    4528 scope.go:122] "RemoveContainer" containerID="d8ed11451a89cd49ea7d668f9d54249605565288576b7cf785b63733dd036e15"
Oct 30 04:19:11.978151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.978125    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:11.983349 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.983318    4528 scope.go:122] "RemoveContainer" containerID="e64057faceec5841c38467923a122fd150c0df758b85da1c7257032d9ad595b5"
Oct 30 04:19:11.991097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:11.991081    4528 scope.go:122] "RemoveContainer" containerID="cb004a8051f03f65add505f6654f0d955c7daf8b76c8fd581127e53b85bd9bb2"
Oct 30 04:19:12.019278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.019256    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:12.031916 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.031892    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:12.058659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.058635    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:19:12.064082 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.064062    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-8tp9s"]
Oct 30 04:19:12.070564 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:12.070539    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-1232cc21: configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:19:12.070679 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:12.070600    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal podName:81616dfd-5ea0-4992-b8c1-eac1536e6336 nodeName:}" failed. No retries permitted until 2026-10-30 04:19:12.570579165 +0000 UTC m=+5910.794828419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal") pod "data-science-collector-collector-0" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336") : configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:19:12.102184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.102115    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:12.211854 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.211751    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:12.575063 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:12.574986    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-1232cc21: configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:19:12.575063 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:12.575058    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal podName:81616dfd-5ea0-4992-b8c1-eac1536e6336 nodeName:}" failed. No retries permitted until 2026-10-30 04:19:13.575042695 +0000 UTC m=+5911.799291963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal") pod "data-science-collector-collector-0" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336") : configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:19:12.951517 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.951485    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:12.969295 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:12.969257    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="81616dfd-5ea0-4992-b8c1-eac1536e6336" containerName="otc-container" containerID="cri-o://fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019" gracePeriod=30
Oct 30 04:19:13.076093 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.076058    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:13.192734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.192700    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:13.228453 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.228432    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:13.281122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281097    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\") pod \"81616dfd-5ea0-4992-b8c1-eac1536e6336\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") "
Oct 30 04:19:13.281122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281125    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\") pod \"81616dfd-5ea0-4992-b8c1-eac1536e6336\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") "
Oct 30 04:19:13.281323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281148    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\" (UniqueName: \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\") pod \"81616dfd-5ea0-4992-b8c1-eac1536e6336\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") "
Oct 30 04:19:13.281323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281186    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\") pod \"81616dfd-5ea0-4992-b8c1-eac1536e6336\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") "
Oct 30 04:19:13.281323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281250    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\" (UniqueName: \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\") pod \"81616dfd-5ea0-4992-b8c1-eac1536e6336\" (UID: \"81616dfd-5ea0-4992-b8c1-eac1536e6336\") "
Oct 30 04:19:13.281727 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.281698    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal" pod "81616dfd-5ea0-4992-b8c1-eac1536e6336" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:13.283926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.283865    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs" pod "81616dfd-5ea0-4992-b8c1-eac1536e6336" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:13.283926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.283889    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert" pod "81616dfd-5ea0-4992-b8c1-eac1536e6336" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:13.283926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.283903    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs" pod "81616dfd-5ea0-4992-b8c1-eac1536e6336" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:13.283926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.283909    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88" pod "81616dfd-5ea0-4992-b8c1-eac1536e6336" (UID: "81616dfd-5ea0-4992-b8c1-eac1536e6336"). InnerVolumeSpecName "kube-api-access-mtq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:13.381881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.381809    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/81616dfd-5ea0-4992-b8c1-eac1536e6336-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:13.381881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.381875    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:13.381881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.381887    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:13.381881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.381899    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtq88\" (UniqueName: \"kubernetes.io/projected/81616dfd-5ea0-4992-b8c1-eac1536e6336-kube-api-access-mtq88\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:13.382161 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.381908    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/81616dfd-5ea0-4992-b8c1-eac1536e6336-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:13.894718 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.894681    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" path="/var/lib/kubelet/pods/1370229c-ea5d-4fcf-8b2c-55196fa676f2/volumes"
Oct 30 04:19:13.895200 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.895175    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" path="/var/lib/kubelet/pods/aa45f3fb-9b72-4a7a-a150-c9263ff6c7af/volumes"
Oct 30 04:19:13.974017 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.973978    4528 generic.go:356] "Generic (PLEG): container finished" podID="81616dfd-5ea0-4992-b8c1-eac1536e6336" containerID="fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019" exitCode=0
Oct 30 04:19:13.974396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.974031    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:13.974396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.974066    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"81616dfd-5ea0-4992-b8c1-eac1536e6336","Type":"ContainerDied","Data":"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"}
Oct 30 04:19:13.974396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.974152    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"81616dfd-5ea0-4992-b8c1-eac1536e6336","Type":"ContainerDied","Data":"2df9742a790f17e7c08280e29582f2670608aff2df61ef4aa37125fdf296e702"}
Oct 30 04:19:13.974396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.974175    4528 scope.go:122] "RemoveContainer" containerID="fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"
Oct 30 04:19:13.982050 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.982032    4528 scope.go:122] "RemoveContainer" containerID="fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"
Oct 30 04:19:13.982327 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:13.982306    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019\": container with ID starting with fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019 not found: ID does not exist" containerID="fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"
Oct 30 04:19:13.982390 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:13.982340    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019"} err="failed to get container status \"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019\": rpc error: code = NotFound desc = could not find container \"fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019\": container with ID starting with fc0d26b459ee5c144374393d7471f85a41d729cdaa7eb084299e4b87611eb019 not found: ID does not exist"
Oct 30 04:19:14.022559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:14.022527    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:14.034015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:14.033989    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:15.892296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:15.892259    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81616dfd-5ea0-4992-b8c1-eac1536e6336" path="/var/lib/kubelet/pods/81616dfd-5ea0-4992-b8c1-eac1536e6336/volumes"
Oct 30 04:19:26.826399 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:26.826341    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:26.978856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:26.978786    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:27.141707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:27.141671    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:27.314985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:27.314938    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:27.555097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:27.555016    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:27.822438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:27.822351    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:27.949445 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:27.949407    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:28.104536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.104444    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:28.671943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.671909    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:28.672489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672464    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="config-reloader"
Oct 30 04:19:28.672489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672490    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="config-reloader"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672514    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" containerName="thanos-querier"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672523    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" containerName="thanos-querier"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672537    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="init-config-reloader"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672545    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="init-config-reloader"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672561    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="prometheus"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672570    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="prometheus"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672583    4528 cpu_manager.go:414] "containerMap: removing container" podUID="81616dfd-5ea0-4992-b8c1-eac1536e6336" containerName="otc-container"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672590    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81616dfd-5ea0-4992-b8c1-eac1536e6336" containerName="otc-container"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672610    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="thanos-sidecar"
Oct 30 04:19:28.672686 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672617    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="thanos-sidecar"
Oct 30 04:19:28.673223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672690    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="config-reloader"
Oct 30 04:19:28.673223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672703    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa45f3fb-9b72-4a7a-a150-c9263ff6c7af" containerName="thanos-querier"
Oct 30 04:19:28.673223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672714    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="prometheus"
Oct 30 04:19:28.673223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672723    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="81616dfd-5ea0-4992-b8c1-eac1536e6336" containerName="otc-container"
Oct 30 04:19:28.673223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.672733    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1370229c-ea5d-4fcf-8b2c-55196fa676f2" containerName="thanos-sidecar"
Oct 30 04:19:28.677343 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.677320    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.692429 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.692403    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.692540 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.692444    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7s6\" (UniqueName: \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.692540 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.692498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.692621 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.692571    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.692621 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.692596    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.694302 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.694275    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:19:28.706234 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.706167    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:19:28.706377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.706344    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\""
Oct 30 04:19:28.708695 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.708674    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:19:28.708879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.708730    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:19:28.708879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.708745    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:19:28.708879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.708750    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:19:28.719023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.719001    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:28.724299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.724168    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:28.793515 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.793484    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.793697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.793526    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.793697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.793578    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.793697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.793601    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7s6\" (UniqueName: \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.793697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.793646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.794377 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.794349    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.796320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.796290    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.796543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.796523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.796915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.796894    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.816509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.816483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7s6\" (UniqueName: \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\") pod \"data-science-collector-collector-0\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:28.859369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.859326    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:28.978413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.978324    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:28.992316 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:28.992278    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:29.289943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:29.289866    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:29.293091 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:19:29.293056    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0efeb16_6f3f_4ad6_bfef_25a99fb9a809.slice/crio-67e21143d0c032fce90c4ab6747f213d24364c02739ce2e5c005a13f15559b87 WatchSource:0}: Error finding container 67e21143d0c032fce90c4ab6747f213d24364c02739ce2e5c005a13f15559b87: Status 404 returned error can't find the container with id 67e21143d0c032fce90c4ab6747f213d24364c02739ce2e5c005a13f15559b87
Oct 30 04:19:29.295042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:29.295012    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:29.295097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:29.295072    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:30.044991 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:30.044956    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809","Type":"ContainerStarted","Data":"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"}
Oct 30 04:19:30.045411 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:30.045008    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809","Type":"ContainerStarted","Data":"67e21143d0c032fce90c4ab6747f213d24364c02739ce2e5c005a13f15559b87"}
Oct 30 04:19:30.085563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:30.085494    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.085479885 podStartE2EDuration="2.085479885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:30.084908503 +0000 UTC m=+5928.309157780" watchObservedRunningTime="2026-10-30 04:19:30.085479885 +0000 UTC m=+5928.309729162"
Oct 30 04:19:31.494403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.494364    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:19:31.497594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.497576    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:19:31.516982 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.516957    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:19:31.530224 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.530191    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:19:31.620578 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.620543    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wn9\" (UniqueName: \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d\" (UID: \"afe6fc13-7439-45dc-8a65-5ed42bbe32cf\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:19:31.708295 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.708258    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:31.721233 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.721200    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wn9\" (UniqueName: \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d\" (UID: \"afe6fc13-7439-45dc-8a65-5ed42bbe32cf\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:19:31.747388 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.747326    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wn9\" (UniqueName: \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d\" (UID: \"afe6fc13-7439-45dc-8a65-5ed42bbe32cf\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:19:31.809195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.809169    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:19:31.959254 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:31.959217    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:32.072427 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.072350    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:32.103290 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.103259    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:19:32.106177 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:19:32.106147    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafe6fc13_7439_45dc_8a65_5ed42bbe32cf.slice/crio-6c2275ca4e81244a85b6902260900bc3344932a3a60ea22eaf286a1cd9635ff4 WatchSource:0}: Error finding container 6c2275ca4e81244a85b6902260900bc3344932a3a60ea22eaf286a1cd9635ff4: Status 404 returned error can't find the container with id 6c2275ca4e81244a85b6902260900bc3344932a3a60ea22eaf286a1cd9635ff4
Oct 30 04:19:32.359775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.359692    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:32.371502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.371472    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:32.375092 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.375069    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.393751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.393727    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:19:32.406460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.406433    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:32.469300 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.469263    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:32.470148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470129    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:19:32.470148 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470130    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:19:32.470268 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470222    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:19:32.470339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470321    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:19:32.470389 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470330    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:19:32.470432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470385    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:19:32.470432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.470406    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-mcw24\""
Oct 30 04:19:32.493868 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.493841    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:19:32.494026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.493925    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:19:32.528050 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528017    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528056    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528164    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528236    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528256    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528274    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528298    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rvz\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528377    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528455 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528851 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.528851 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.528486    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.628881 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.628849    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629106 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.628892    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629106 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.628915    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rvz\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629175    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629258    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629288    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629337    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629402    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629564 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629433    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.629805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.629780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.632363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.632334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.632492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.632404    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.632873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.632790    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.633030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.632948    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.633109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.633027    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.633196 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.633177    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.633711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.633680    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.633944 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.633927    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.634423 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.634402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.634666 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.634649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:32.652306 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:32.652279    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rvz\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:33.060007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:33.059919    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" event={"ID":"afe6fc13-7439-45dc-8a65-5ed42bbe32cf","Type":"ContainerStarted","Data":"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"}
Oct 30 04:19:33.060007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:33.059965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" event={"ID":"afe6fc13-7439-45dc-8a65-5ed42bbe32cf","Type":"ContainerStarted","Data":"6c2275ca4e81244a85b6902260900bc3344932a3a60ea22eaf286a1cd9635ff4"}
Oct 30 04:19:33.098197 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:33.098146    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" podStartSLOduration=2.098132187 podStartE2EDuration="2.098132187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:33.097190968 +0000 UTC m=+5931.321440227" watchObservedRunningTime="2026-10-30 04:19:33.098132187 +0000 UTC m=+5931.322381458"
Oct 30 04:19:34.630725 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:34.630686    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:19:35.350890 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.350853    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.378340 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.378310    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.451194 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.451155    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.451336 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.451302    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.464956 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.464934    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.562466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.562439    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.589254 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.589225    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:35.696524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:35.696486    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:36.027378 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:36.027315    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:19:36.030112 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:19:36.030084    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4187c128_6ae7_4bf5_a8ea_2b16933719d2.slice/crio-4d25f95310bee19034616d44a74e7d4514a783cd639466d7f351f62029b5f7b6 WatchSource:0}: Error finding container 4d25f95310bee19034616d44a74e7d4514a783cd639466d7f351f62029b5f7b6: Status 404 returned error can't find the container with id 4d25f95310bee19034616d44a74e7d4514a783cd639466d7f351f62029b5f7b6
Oct 30 04:19:36.073811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:36.073786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerStarted","Data":"4d25f95310bee19034616d44a74e7d4514a783cd639466d7f351f62029b5f7b6"}
Oct 30 04:19:37.079871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:37.079837    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerStarted","Data":"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"}
Oct 30 04:19:40.092432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:40.092403    4528 generic.go:356] "Generic (PLEG): container finished" podID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0" exitCode=0
Oct 30 04:19:40.092798 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:40.092479    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerDied","Data":"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"}
Oct 30 04:19:41.098999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:41.098960    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerStarted","Data":"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"}
Oct 30 04:19:41.099421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:41.099029    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerStarted","Data":"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"}
Oct 30 04:19:41.099421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:41.099053    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerStarted","Data":"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"}
Oct 30 04:19:41.173867 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:41.173794    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=5.587661592 podStartE2EDuration="9.173780476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586118884s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:41.172748073 +0000 UTC m=+5939.396997351" watchObservedRunningTime="2026-10-30 04:19:41.173780476 +0000 UTC m=+5939.398029747"
Oct 30 04:19:41.891204 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:41.891167    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:45.697044 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:45.697011    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:47.989435 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:47.989396    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:48.188529 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:48.188488    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:48.301883 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:48.301796    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:49.889713 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:49.889674    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:50.004845 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:50.004795    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:50.127691 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:50.127662    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:19:50.697326 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:50.697290    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:50.702562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:50.702539    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:51.143475 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:51.143445    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:19:53.666915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:53.666876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:53.667357 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:53.667130    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" containerName="otc-container" containerID="cri-o://d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb" gracePeriod=30
Oct 30 04:19:53.911855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:53.911814    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.002310 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002229    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\") pod \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") "
Oct 30 04:19:54.002310 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002278    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\") pod \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") "
Oct 30 04:19:54.002310 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002302    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\") pod \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") "
Oct 30 04:19:54.002559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002334    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\") pod \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") "
Oct 30 04:19:54.002559 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002409    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\" (UniqueName: \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\") pod \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\" (UID: \"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809\") "
Oct 30 04:19:54.002756 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.002728    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal" pod "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" (UID: "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:19:54.004499 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.004471    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs" pod "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" (UID: "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:54.004617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.004512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6" pod "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" (UID: "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809"). InnerVolumeSpecName "kube-api-access-zk7s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:19:54.004617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.004512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs" pod "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" (UID: "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:54.004617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.004560    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert" pod "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" (UID: "b0efeb16-6f3f-4ad6-bfef-25a99fb9a809"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:19:54.103601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.103568    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7s6\" (UniqueName: \"kubernetes.io/projected/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-kube-api-access-zk7s6\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:54.103601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.103596    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:54.103601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.103606    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:54.103866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.103615    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:54.103866 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.103625    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:19:54.153432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.153397    4528 generic.go:356] "Generic (PLEG): container finished" podID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" containerID="d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb" exitCode=0
Oct 30 04:19:54.153584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.153443    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.153584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.153455    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809","Type":"ContainerDied","Data":"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"}
Oct 30 04:19:54.153584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.153496    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b0efeb16-6f3f-4ad6-bfef-25a99fb9a809","Type":"ContainerDied","Data":"67e21143d0c032fce90c4ab6747f213d24364c02739ce2e5c005a13f15559b87"}
Oct 30 04:19:54.153584 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.153519    4528 scope.go:122] "RemoveContainer" containerID="d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"
Oct 30 04:19:54.161955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.161937    4528 scope.go:122] "RemoveContainer" containerID="d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"
Oct 30 04:19:54.162199 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:19:54.162180    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb\": container with ID starting with d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb not found: ID does not exist" containerID="d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"
Oct 30 04:19:54.162253 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.162207    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb"} err="failed to get container status \"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb\": rpc error: code = NotFound desc = could not find container \"d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb\": container with ID starting with d9f83e2324d49e153f52230fd27c10d6a8b496a094dd9d85a6b804fe230547eb not found: ID does not exist"
Oct 30 04:19:54.202521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.202491    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:54.206762 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.206739    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:54.249033 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.249002    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:54.249330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.249317    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" containerName="otc-container"
Oct 30 04:19:54.249374 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.249332    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" containerName="otc-container"
Oct 30 04:19:54.249414 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.249409    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" containerName="otc-container"
Oct 30 04:19:54.253719 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.253678    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.268207 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.268178    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\""
Oct 30 04:19:54.268325 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.268223    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:19:54.268374 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.268328    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:19:54.271108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.271089    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:19:54.275516 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.275493    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:54.305379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.305351    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.305534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.305392    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.305534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.305424    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.305534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.305508    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.305534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.305529    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.368314 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.368282    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:19:54.406673 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.406646    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.406787 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.406677    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.406875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.406808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.406918 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.406881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.406971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.406919    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.407404 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.407380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.409155 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.409132    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.409265 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.409229    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.409265 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.409241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.429653 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.429621    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\") pod \"data-science-collector-collector-0\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.565910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.565802    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:19:54.861948 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.861874    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:19:54.865184 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:19:54.865160    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf127aa4d_f097_44e2_bc2f_8d3ff1a33a5a.slice/crio-440d625958bd35393a6962c44b12c3205e5dde1d11be26f42425dcd67c024eff WatchSource:0}: Error finding container 440d625958bd35393a6962c44b12c3205e5dde1d11be26f42425dcd67c024eff: Status 404 returned error can't find the container with id 440d625958bd35393a6962c44b12c3205e5dde1d11be26f42425dcd67c024eff
Oct 30 04:19:54.867491 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.867460    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:54.867600 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:54.867520    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:19:55.161228 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:55.161193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a","Type":"ContainerStarted","Data":"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"}
Oct 30 04:19:55.161410 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:55.161240    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a","Type":"ContainerStarted","Data":"440d625958bd35393a6962c44b12c3205e5dde1d11be26f42425dcd67c024eff"}
Oct 30 04:19:55.202067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:55.202017    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.201999256 podStartE2EDuration="1.201999256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:55.201960569 +0000 UTC m=+5953.426209847" watchObservedRunningTime="2026-10-30 04:19:55.201999256 +0000 UTC m=+5953.426248530"
Oct 30 04:19:55.893166 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:19:55.893134    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0efeb16-6f3f-4ad6-bfef-25a99fb9a809" path="/var/lib/kubelet/pods/b0efeb16-6f3f-4ad6-bfef-25a99fb9a809/volumes"
Oct 30 04:20:03.636731 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:03.636693    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:03.747954 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:03.747918    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:03.950885 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:03.950780    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:04.064891 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:04.064861    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:04.631006 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:04.630971    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:20:05.599130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:05.599096    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:05.723070 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:05.723034    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:05.837240 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:05.837199    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:08.750680 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:08.750645    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:08.751205 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:08.750906    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" containerName="otc-container" containerID="cri-o://98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17" gracePeriod=30
Oct 30 04:20:08.988707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:08.988682    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.029736 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.029650    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\") pod \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") "
Oct 30 04:20:09.029736 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.029722    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\") pod \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") "
Oct 30 04:20:09.029970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.029743    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\") pod \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") "
Oct 30 04:20:09.029970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.029883    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\") pod \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") "
Oct 30 04:20:09.029970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.029934    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\") pod \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\" (UID: \"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a\") "
Oct 30 04:20:09.030481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.030115    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal" pod "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" (UID: "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:09.030481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.030213    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:09.032613 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.032583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert" pod "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" (UID: "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:09.032906 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.032866    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs" pod "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" (UID: "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:09.033013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.032992    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4" pod "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" (UID: "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a"). InnerVolumeSpecName "kube-api-access-qszz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:09.033174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.033149    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs" pod "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" (UID: "f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:09.130745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.130694    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:09.130745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.130742    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:09.130745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.130754    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-kube-api-access-qszz4\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:09.131031 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.130765    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:09.217229 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.217193    4528 generic.go:356] "Generic (PLEG): container finished" podID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" containerID="98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17" exitCode=0
Oct 30 04:20:09.217380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.217242    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.217380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.217274    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a","Type":"ContainerDied","Data":"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"}
Oct 30 04:20:09.217380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.217326    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a","Type":"ContainerDied","Data":"440d625958bd35393a6962c44b12c3205e5dde1d11be26f42425dcd67c024eff"}
Oct 30 04:20:09.217380 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.217348    4528 scope.go:122] "RemoveContainer" containerID="98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"
Oct 30 04:20:09.225661 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.225637    4528 scope.go:122] "RemoveContainer" containerID="98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"
Oct 30 04:20:09.225934 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:09.225913    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17\": container with ID starting with 98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17 not found: ID does not exist" containerID="98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"
Oct 30 04:20:09.225993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.225942    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17"} err="failed to get container status \"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17\": rpc error: code = NotFound desc = could not find container \"98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17\": container with ID starting with 98e0fcfdfa4dabec94945241d3a8298e3ae9b6851566d850482ee2e7fda0ca17 not found: ID does not exist"
Oct 30 04:20:09.267274 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.267233    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:09.271623 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.271595    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:09.311628 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.311556    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:09.311861 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.311848    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" containerName="otc-container"
Oct 30 04:20:09.311913 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.311861    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" containerName="otc-container"
Oct 30 04:20:09.311950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.311928    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" containerName="otc-container"
Oct 30 04:20:09.316136 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.316119    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.336283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.336252    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:09.391975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.391940    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:20:09.411502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.411469    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:20:09.411502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.411503    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:20:09.411663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.411508    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wvqg5\""
Oct 30 04:20:09.411663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.411505    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:20:09.432125 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.432097    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5j4d\" (UniqueName: \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.432179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.432129    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.432179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.432154    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.432337 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.432313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.432394 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.432380    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.533417 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.533371    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.533536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.533438    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.533536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.533475    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.533536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.533512    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5j4d\" (UniqueName: \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.533536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.533528    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.534270 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.534241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.535963 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.535942    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.536048 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.535999    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.536108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.536091    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.556248 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.556224    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5j4d\" (UniqueName: \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\") pod \"data-science-collector-collector-0\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.627760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.627696    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:09.891655 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.891622    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a" path="/var/lib/kubelet/pods/f127aa4d-f097-44e2-bc2f-8d3ff1a33a5a/volumes"
Oct 30 04:20:09.927313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.927279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:09.931041 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:09.931014    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbe2311_2c60_416e_8fe6_a76e3e4763e8.slice/crio-6f53c115e6124d3f262c806d09ff81d70b25f6d8da8f65d373492ce2f2ccf943 WatchSource:0}: Error finding container 6f53c115e6124d3f262c806d09ff81d70b25f6d8da8f65d373492ce2f2ccf943: Status 404 returned error can't find the container with id 6f53c115e6124d3f262c806d09ff81d70b25f6d8da8f65d373492ce2f2ccf943
Oct 30 04:20:09.933015 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.932980    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:09.933100 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:09.933042    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:10.224718 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.224628    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbe2311-2c60-416e-8fe6-a76e3e4763e8","Type":"ContainerStarted","Data":"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"}
Oct 30 04:20:10.224718 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.224677    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbe2311-2c60-416e-8fe6-a76e3e4763e8","Type":"ContainerStarted","Data":"6f53c115e6124d3f262c806d09ff81d70b25f6d8da8f65d373492ce2f2ccf943"}
Oct 30 04:20:10.268684 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.268630    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.268614259 podStartE2EDuration="1.268614259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:10.267121051 +0000 UTC m=+5968.491370328" watchObservedRunningTime="2026-10-30 04:20:10.268614259 +0000 UTC m=+5968.492863526"
Oct 30 04:20:10.330585 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.330547    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:20:10.330805 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.330763    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" podUID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" containerName="thanos-querier" containerID="cri-o://766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48" gracePeriod=30
Oct 30 04:20:10.401796 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.401741    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:10.512246 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.511834    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:10.512246 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.512149    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="prometheus" containerID="cri-o://f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558" gracePeriod=600
Oct 30 04:20:10.512553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.512523    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="thanos-sidecar" containerID="cri-o://b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4" gracePeriod=600
Oct 30 04:20:10.512669 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.512648    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="config-reloader" containerID="cri-o://0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652" gracePeriod=600
Oct 30 04:20:10.536215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.536130    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:10.628286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.628239    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:20:10.653446 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.653406    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:10.745575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.745545    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\" (UniqueName: \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\") pod \"afe6fc13-7439-45dc-8a65-5ed42bbe32cf\" (UID: \"afe6fc13-7439-45dc-8a65-5ed42bbe32cf\") "
Oct 30 04:20:10.747531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.747504    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9" pod "afe6fc13-7439-45dc-8a65-5ed42bbe32cf" (UID: "afe6fc13-7439-45dc-8a65-5ed42bbe32cf"). InnerVolumeSpecName "kube-api-access-g2wn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:10.763620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.763566    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:10.791503 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.791471    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:10.847107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847073    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847107 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847117    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847188    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847219    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847250    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847284    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847514    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847571    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847612    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847644    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\" (UID: \"4187c128-6ae7-4bf5-a8ea-2b16933719d2\") "
Oct 30 04:20:10.847932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847644    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:10.847932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847915    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.848098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847937    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wn9\" (UniqueName: \"kubernetes.io/projected/afe6fc13-7439-45dc-8a65-5ed42bbe32cf-kube-api-access-g2wn9\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.848098 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.847995    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:10.848464 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.848395    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:10.852076 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852048    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.852188 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852070    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.852188 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852126    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.852188 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852169    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.852557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852518    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.852879 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852852    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:20:10.852988 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.852946    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "kube-api-access-62rvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:10.856647 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.856626    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:10.856965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.856944    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:20:10.866440 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.866306    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config" pod "4187c128-6ae7-4bf5-a8ea-2b16933719d2" (UID: "4187c128-6ae7-4bf5-a8ea-2b16933719d2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:10.910208 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.910172    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:10.949339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949296    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949326    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949339 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949336    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949347    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949364    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949373    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62rvz\" (UniqueName: \"kubernetes.io/projected/4187c128-6ae7-4bf5-a8ea-2b16933719d2-kube-api-access-62rvz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949383    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949394    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949403    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4187c128-6ae7-4bf5-a8ea-2b16933719d2-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949412    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4187c128-6ae7-4bf5-a8ea-2b16933719d2-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949421    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4187c128-6ae7-4bf5-a8ea-2b16933719d2-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:10.949575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.949447    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:20:10.960026 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:10.960006    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:20:11.050160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.050082    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:20:11.148452 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.148419    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:11.205170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.205074    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:11.229755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.229726    4528 generic.go:356] "Generic (PLEG): container finished" podID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" containerID="766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48" exitCode=0
Oct 30 04:20:11.229946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.229773    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"
Oct 30 04:20:11.229946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.229811    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" event={"ID":"afe6fc13-7439-45dc-8a65-5ed42bbe32cf","Type":"ContainerDied","Data":"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"}
Oct 30 04:20:11.229946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.229871    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d" event={"ID":"afe6fc13-7439-45dc-8a65-5ed42bbe32cf","Type":"ContainerDied","Data":"6c2275ca4e81244a85b6902260900bc3344932a3a60ea22eaf286a1cd9635ff4"}
Oct 30 04:20:11.229946 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.229892    4528 scope.go:122] "RemoveContainer" containerID="766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"
Oct 30 04:20:11.232179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232156    4528 generic.go:356] "Generic (PLEG): container finished" podID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4" exitCode=0
Oct 30 04:20:11.232179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232180    4528 generic.go:356] "Generic (PLEG): container finished" podID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652" exitCode=0
Oct 30 04:20:11.232353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232188    4528 generic.go:356] "Generic (PLEG): container finished" podID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558" exitCode=0
Oct 30 04:20:11.232353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232212    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:11.232353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232240    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerDied","Data":"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"}
Oct 30 04:20:11.232353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232305    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerDied","Data":"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"}
Oct 30 04:20:11.232353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerDied","Data":"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"}
Oct 30 04:20:11.232528 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232385    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4187c128-6ae7-4bf5-a8ea-2b16933719d2","Type":"ContainerDied","Data":"4d25f95310bee19034616d44a74e7d4514a783cd639466d7f351f62029b5f7b6"}
Oct 30 04:20:11.232967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.232935    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-collector-collector-0" secret="" err="secret \"data-science-collector-collector-dockercfg-wvqg5\" not found"
Oct 30 04:20:11.238902 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.238883    4528 scope.go:122] "RemoveContainer" containerID="766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"
Oct 30 04:20:11.239214 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.239190    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48\": container with ID starting with 766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48 not found: ID does not exist" containerID="766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"
Oct 30 04:20:11.239316 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.239219    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48"} err="failed to get container status \"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48\": rpc error: code = NotFound desc = could not find container \"766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48\": container with ID starting with 766b7fff8445b7518a18584b7e92caa59eb9ec302fcde09d3ac789ad753a4d48 not found: ID does not exist"
Oct 30 04:20:11.239316 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.239236    4528 scope.go:122] "RemoveContainer" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"
Oct 30 04:20:11.247596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.247573    4528 scope.go:122] "RemoveContainer" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"
Oct 30 04:20:11.258718 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.258692    4528 scope.go:122] "RemoveContainer" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"
Oct 30 04:20:11.268214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.268186    4528 scope.go:122] "RemoveContainer" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"
Oct 30 04:20:11.275935 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.275900    4528 scope.go:122] "RemoveContainer" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"
Oct 30 04:20:11.276192 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.276170    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": container with ID starting with b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4 not found: ID does not exist" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"
Oct 30 04:20:11.276272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276204    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"} err="failed to get container status \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": rpc error: code = NotFound desc = could not find container \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": container with ID starting with b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4 not found: ID does not exist"
Oct 30 04:20:11.276272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276221    4528 scope.go:122] "RemoveContainer" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"
Oct 30 04:20:11.276470 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.276449    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": container with ID starting with 0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652 not found: ID does not exist" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"
Oct 30 04:20:11.276522 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276475    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"} err="failed to get container status \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": rpc error: code = NotFound desc = could not find container \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": container with ID starting with 0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652 not found: ID does not exist"
Oct 30 04:20:11.276522 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276488    4528 scope.go:122] "RemoveContainer" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"
Oct 30 04:20:11.276704 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.276689    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": container with ID starting with f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558 not found: ID does not exist" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"
Oct 30 04:20:11.276762 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276707    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"} err="failed to get container status \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": rpc error: code = NotFound desc = could not find container \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": container with ID starting with f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558 not found: ID does not exist"
Oct 30 04:20:11.276762 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.276719    4528 scope.go:122] "RemoveContainer" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"
Oct 30 04:20:11.277036 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.277020    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": container with ID starting with 611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0 not found: ID does not exist" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"
Oct 30 04:20:11.277089 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277037    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"} err="failed to get container status \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": rpc error: code = NotFound desc = could not find container \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": container with ID starting with 611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0 not found: ID does not exist"
Oct 30 04:20:11.277089 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277056    4528 scope.go:122] "RemoveContainer" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"
Oct 30 04:20:11.277251 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277233    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"} err="failed to get container status \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": rpc error: code = NotFound desc = could not find container \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": container with ID starting with b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4 not found: ID does not exist"
Oct 30 04:20:11.277251 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277250    4528 scope.go:122] "RemoveContainer" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"
Oct 30 04:20:11.277488 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277457    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"} err="failed to get container status \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": rpc error: code = NotFound desc = could not find container \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": container with ID starting with 0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652 not found: ID does not exist"
Oct 30 04:20:11.277551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277491    4528 scope.go:122] "RemoveContainer" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"
Oct 30 04:20:11.277725 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277705    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"} err="failed to get container status \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": rpc error: code = NotFound desc = could not find container \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": container with ID starting with f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558 not found: ID does not exist"
Oct 30 04:20:11.277725 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277724    4528 scope.go:122] "RemoveContainer" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"
Oct 30 04:20:11.277972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277955    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"} err="failed to get container status \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": rpc error: code = NotFound desc = could not find container \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": container with ID starting with 611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0 not found: ID does not exist"
Oct 30 04:20:11.277972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.277970    4528 scope.go:122] "RemoveContainer" containerID="b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"
Oct 30 04:20:11.278210 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278189    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4"} err="failed to get container status \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": rpc error: code = NotFound desc = could not find container \"b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4\": container with ID starting with b689a6e5a3133b6c88b563cbdfb70269a1b500ee06333cda09557b47f512f7e4 not found: ID does not exist"
Oct 30 04:20:11.278210 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278209    4528 scope.go:122] "RemoveContainer" containerID="0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"
Oct 30 04:20:11.278481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278461    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652"} err="failed to get container status \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": rpc error: code = NotFound desc = could not find container \"0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652\": container with ID starting with 0c432db9374f40ab4b8a0f149c90ba19a9960c75f5d4667a3fb46034552c1652 not found: ID does not exist"
Oct 30 04:20:11.278481 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278480    4528 scope.go:122] "RemoveContainer" containerID="f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"
Oct 30 04:20:11.278679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278660    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558"} err="failed to get container status \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": rpc error: code = NotFound desc = could not find container \"f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558\": container with ID starting with f30b33597fbf3a4c4338620ed941f7711c6530e91774669d09c396f70aa85558 not found: ID does not exist"
Oct 30 04:20:11.278744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278678    4528 scope.go:122] "RemoveContainer" containerID="611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"
Oct 30 04:20:11.278920 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.278896    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0"} err="failed to get container status \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": rpc error: code = NotFound desc = could not find container \"611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0\": container with ID starting with 611f883c1664760e84ba996b75a3b37a65c50f94dc1fce325b97aa9b95a30cc0 not found: ID does not exist"
Oct 30 04:20:11.319570 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.319494    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:11.336260 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.336225    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:11.352332 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.352298    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-1232cc21: configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:20:11.352485 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.352377    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal podName:5cbe2311-2c60-416e-8fe6-a76e3e4763e8 nodeName:}" failed. No retries permitted until 2026-10-30 04:20:11.852355373 +0000 UTC m=+5970.076604645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal") pod "data-science-collector-collector-0" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8") : configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:20:11.353855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.353802    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:11.381728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.381696    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:20:11.396597 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.396566    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dpl6d"]
Oct 30 04:20:11.474053 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.474020    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:11.857109 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.857076    4528 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-collector-collector-1232cc21: configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:20:11.857260 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:11.857142    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal podName:5cbe2311-2c60-416e-8fe6-a76e3e4763e8 nodeName:}" failed. No retries permitted until 2026-10-30 04:20:12.85712921 +0000 UTC m=+5971.081378478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal") pod "data-science-collector-collector-0" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8") : configmap "data-science-collector-collector-1232cc21" not found
Oct 30 04:20:11.891986 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.891954    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" path="/var/lib/kubelet/pods/4187c128-6ae7-4bf5-a8ea-2b16933719d2/volumes"
Oct 30 04:20:11.892413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:11.892399    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" path="/var/lib/kubelet/pods/afe6fc13-7439-45dc-8a65-5ed42bbe32cf/volumes"
Oct 30 04:20:12.217335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.217302    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:12.237954 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.237913    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" containerName="otc-container" containerID="cri-o://1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384" gracePeriod=30
Oct 30 04:20:12.339492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.339411    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:12.453645 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.453609    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:12.478173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.478120    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:12.563599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.563570    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\" (UniqueName: \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\") pod \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") "
Oct 30 04:20:12.563755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.563621    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\") pod \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") "
Oct 30 04:20:12.563755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.563666    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\") pod \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") "
Oct 30 04:20:12.563755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.563689    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\") pod \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") "
Oct 30 04:20:12.563755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.563717    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\") pod \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\" (UID: \"5cbe2311-2c60-416e-8fe6-a76e3e4763e8\") "
Oct 30 04:20:12.564081 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.564048    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal" pod "5cbe2311-2c60-416e-8fe6-a76e3e4763e8" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:12.565975 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.565946    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs" pod "5cbe2311-2c60-416e-8fe6-a76e3e4763e8" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:12.566062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.565955    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d" pod "5cbe2311-2c60-416e-8fe6-a76e3e4763e8" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8"). InnerVolumeSpecName "kube-api-access-x5j4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:12.566062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.565956    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert" pod "5cbe2311-2c60-416e-8fe6-a76e3e4763e8" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:12.566062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.565982    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs" pod "5cbe2311-2c60-416e-8fe6-a76e3e4763e8" (UID: "5cbe2311-2c60-416e-8fe6-a76e3e4763e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:12.664720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.664689    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:12.664720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.664715    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5j4d\" (UniqueName: \"kubernetes.io/projected/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-kube-api-access-x5j4d\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:12.664720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.664727    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:12.664978 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.664736    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:12.664978 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:12.664745    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbe2311-2c60-416e-8fe6-a76e3e4763e8-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:13.243052 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.243018    4528 generic.go:356] "Generic (PLEG): container finished" podID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" containerID="1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384" exitCode=0
Oct 30 04:20:13.243537 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.243061    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:13.243537 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.243103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbe2311-2c60-416e-8fe6-a76e3e4763e8","Type":"ContainerDied","Data":"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"}
Oct 30 04:20:13.243537 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.243156    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbe2311-2c60-416e-8fe6-a76e3e4763e8","Type":"ContainerDied","Data":"6f53c115e6124d3f262c806d09ff81d70b25f6d8da8f65d373492ce2f2ccf943"}
Oct 30 04:20:13.243537 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.243188    4528 scope.go:122] "RemoveContainer" containerID="1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"
Oct 30 04:20:13.251384 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.251369    4528 scope.go:122] "RemoveContainer" containerID="1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"
Oct 30 04:20:13.251635 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:13.251619    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384\": container with ID starting with 1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384 not found: ID does not exist" containerID="1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"
Oct 30 04:20:13.251679 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.251646    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384"} err="failed to get container status \"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384\": rpc error: code = NotFound desc = could not find container \"1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384\": container with ID starting with 1ada0d51366bfb92116a46cef9ea34a8cde0ecf3c30b7ddded035ff7f8bfd384 not found: ID does not exist"
Oct 30 04:20:13.303732 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.303700    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:13.315141 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.315112    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:13.892421 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:13.892382    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" path="/var/lib/kubelet/pods/5cbe2311-2c60-416e-8fe6-a76e3e4763e8/volumes"
Oct 30 04:20:16.827459 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:16.827425    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:16.950418 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:16.950381    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.077425 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.077391    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.211225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.211163    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.328739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.328707    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.484011 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.483926    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.763910 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.763813    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:17.764117 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764105    4528 cpu_manager.go:414] "containerMap: removing container" podUID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" containerName="thanos-querier"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764118    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" containerName="thanos-querier"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764127    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="init-config-reloader"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764132    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="init-config-reloader"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764146    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="prometheus"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764152    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="prometheus"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764160    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" containerName="otc-container"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764164    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" containerName="otc-container"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764171    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="config-reloader"
Oct 30 04:20:17.764174 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764176    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="config-reloader"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764182    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="thanos-sidecar"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764186    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="thanos-sidecar"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764230    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe6fc13-7439-45dc-8a65-5ed42bbe32cf" containerName="thanos-querier"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764240    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="prometheus"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764246    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbe2311-2c60-416e-8fe6-a76e3e4763e8" containerName="otc-container"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764252    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="config-reloader"
Oct 30 04:20:17.764496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.764257    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4187c128-6ae7-4bf5-a8ea-2b16933719d2" containerName="thanos-sidecar"
Oct 30 04:20:17.768362 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.768341    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:17.785042 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.785015    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:20:17.796782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.796749    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:20:17.796940 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.796919    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:20:17.797882 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.797863    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:17.864601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.864556    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:17.908183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:17.908153    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjck\" (UniqueName: \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5\" (UID: \"745aa4c9-1934-4912-9757-5e6de52dd2c1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:18.009531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.009496    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjck\" (UniqueName: \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5\" (UID: \"745aa4c9-1934-4912-9757-5e6de52dd2c1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:18.032637 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.032573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjck\" (UniqueName: \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\") pod \"thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5\" (UID: \"745aa4c9-1934-4912-9757-5e6de52dd2c1\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:18.055927 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.055893    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:18.080541 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.080504    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:18.223004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.222967    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:18.377596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.377522    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:18.380615 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:18.380583    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod745aa4c9_1934_4912_9757_5e6de52dd2c1.slice/crio-3d3a772893b9ade94964d48226056e94777bd708f754606b091223ef4bf220a5 WatchSource:0}: Error finding container 3d3a772893b9ade94964d48226056e94777bd708f754606b091223ef4bf220a5: Status 404 returned error can't find the container with id 3d3a772893b9ade94964d48226056e94777bd708f754606b091223ef4bf220a5
Oct 30 04:20:18.770854 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.770122    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:18.773454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.773432    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:18.777088 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.777059    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:18.800830 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.800790    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:20:18.802521 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.802493    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:18.872046 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.872012    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:20:18.872046 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.872042    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:20:18.872461 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.872117    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\""
Oct 30 04:20:18.872461 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.872158    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:20:18.899998 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.899970    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:18.915157 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.915131    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:18.915299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.915181    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62kf\" (UniqueName: \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:18.915299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.915202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:18.915379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.915320    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:18.915379 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:18.915340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.016276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.016240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62kf\" (UniqueName: \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.016276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.016276    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.016579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.016310    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.016579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.016483    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.016579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.016542    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.017236 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.017209    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.018974 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.018950    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.019108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.018986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.019108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.019035    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.019663 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.019637    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:19.039624 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.039560    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62kf\" (UniqueName: \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\") pod \"data-science-collector-collector-1\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.089892 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.089856    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:19.138095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.137938    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:19.276320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.276286    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" event={"ID":"745aa4c9-1934-4912-9757-5e6de52dd2c1","Type":"ContainerStarted","Data":"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"}
Oct 30 04:20:19.276320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.276325    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" event={"ID":"745aa4c9-1934-4912-9757-5e6de52dd2c1","Type":"ContainerStarted","Data":"3d3a772893b9ade94964d48226056e94777bd708f754606b091223ef4bf220a5"}
Oct 30 04:20:19.313956 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.313851    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" podStartSLOduration=2.313834473 podStartE2EDuration="2.313834473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:19.31339616 +0000 UTC m=+5977.537645416" watchObservedRunningTime="2026-10-30 04:20:19.313834473 +0000 UTC m=+5977.538083750"
Oct 30 04:20:19.385309 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.385277    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:19.388758 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:19.388731    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d761ac_9500_46a5_a472_b3c24baffdd2.slice/crio-ef0b7de14d0c9f46d6d2fea5c06e4a3ca34d2bc0f2121d89d175a0064799e2c0 WatchSource:0}: Error finding container ef0b7de14d0c9f46d6d2fea5c06e4a3ca34d2bc0f2121d89d175a0064799e2c0: Status 404 returned error can't find the container with id ef0b7de14d0c9f46d6d2fea5c06e4a3ca34d2bc0f2121d89d175a0064799e2c0
Oct 30 04:20:19.391019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.390990    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:19.391094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:19.391050    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:20.282193 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:20.282153    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"71d761ac-9500-46a5-a472-b3c24baffdd2","Type":"ContainerStarted","Data":"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"}
Oct 30 04:20:20.282582 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:20.282234    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"71d761ac-9500-46a5-a472-b3c24baffdd2","Type":"ContainerStarted","Data":"ef0b7de14d0c9f46d6d2fea5c06e4a3ca34d2bc0f2121d89d175a0064799e2c0"}
Oct 30 04:20:20.323464 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:20.323314    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.323299857 podStartE2EDuration="2.323299857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:20.32310001 +0000 UTC m=+5978.547349289" watchObservedRunningTime="2026-10-30 04:20:20.323299857 +0000 UTC m=+5978.547549135"
Oct 30 04:20:21.495190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.495138    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:20:21.498304 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.498288    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:21.518320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.518296    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:20:21.638122 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.638086    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hwz\" (UniqueName: \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb\" (UID: \"0b432647-4f35-49b6-b102-f65b0fb95836\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:21.696563 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.696528    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:21.739516 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.739486    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hwz\" (UniqueName: \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb\" (UID: \"0b432647-4f35-49b6-b102-f65b0fb95836\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:21.762616 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.762556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hwz\" (UniqueName: \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb\" (UID: \"0b432647-4f35-49b6-b102-f65b0fb95836\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:21.809710 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.809687    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:21.934457 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:21.934422    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:22.064035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.063954    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:22.105942 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.105899    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:20:22.109298 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:22.109268    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b432647_4f35_49b6_b102_f65b0fb95836.slice/crio-0eea3d78bab1da5f8488434d2bee0c6a36ca040e3b24f55e84b31bb80a56ef6f WatchSource:0}: Error finding container 0eea3d78bab1da5f8488434d2bee0c6a36ca040e3b24f55e84b31bb80a56ef6f: Status 404 returned error can't find the container with id 0eea3d78bab1da5f8488434d2bee0c6a36ca040e3b24f55e84b31bb80a56ef6f
Oct 30 04:20:22.292457 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.292422    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" event={"ID":"0b432647-4f35-49b6-b102-f65b0fb95836","Type":"ContainerStarted","Data":"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"}
Oct 30 04:20:22.292457 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.292462    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" event={"ID":"0b432647-4f35-49b6-b102-f65b0fb95836","Type":"ContainerStarted","Data":"0eea3d78bab1da5f8488434d2bee0c6a36ca040e3b24f55e84b31bb80a56ef6f"}
Oct 30 04:20:22.331720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.331612    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" podStartSLOduration=1.331595608 podStartE2EDuration="1.331595608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:22.330580574 +0000 UTC m=+5980.554829851" watchObservedRunningTime="2026-10-30 04:20:22.331595608 +0000 UTC m=+5980.555844886"
Oct 30 04:20:22.364278 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.364245    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:22.364497 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.364470    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" podUID="745aa4c9-1934-4912-9757-5e6de52dd2c1" containerName="thanos-querier" containerID="cri-o://2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632" gracePeriod=30
Oct 30 04:20:22.366165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.366068    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:22.434454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.434424    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:22.437947 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.437902    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.452450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.452428    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:20:22.452550 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.452449    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:20:22.452695 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.452680    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:20:22.466720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.466697    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:22.475929 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.475910    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:20:22.476200 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476178    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:20:22.476327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476196    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:20:22.476327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476236    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:20:22.476327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476239    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:20:22.476327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476262    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:20:22.476327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.476281    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-fsr8m\""
Oct 30 04:20:22.510935 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.510894    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:22.546309 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546280    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546444 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546336    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546542 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546510    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546622 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546603    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546702 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546682    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546745 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546792 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546761    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546859 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546843    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.546976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546955    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.547012 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.546997    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.547055 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.547041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7xz\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.596280 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.596257    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:22.647358 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.647332    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\" (UniqueName: \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\") pod \"745aa4c9-1934-4912-9757-5e6de52dd2c1\" (UID: \"745aa4c9-1934-4912-9757-5e6de52dd2c1\") "
Oct 30 04:20:22.647491 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.647479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.647546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.647502    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.647546 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.647524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.647659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.647553    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648058 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648250 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648223    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648354 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648307    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648429 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7xz\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648489 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648451    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648550 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648483    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648550 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648685 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648572    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648685 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648617    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648685 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648668    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.648873 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.648483    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.651690 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651665    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.651690 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651672    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.651901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651745    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck" pod "745aa4c9-1934-4912-9757-5e6de52dd2c1" (UID: "745aa4c9-1934-4912-9757-5e6de52dd2c1"). InnerVolumeSpecName "kube-api-access-5fjck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:22.651901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651764    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.651901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651794    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.651901 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.651885    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.652137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.652119    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.652285 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.652263    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.653215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.653195    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.670808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.670785    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7xz\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:22.749993 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:22.749962    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fjck\" (UniqueName: \"kubernetes.io/projected/745aa4c9-1934-4912-9757-5e6de52dd2c1-kube-api-access-5fjck\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:23.298876 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.298833    4528 generic.go:356] "Generic (PLEG): container finished" podID="745aa4c9-1934-4912-9757-5e6de52dd2c1" containerID="2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632" exitCode=0
Oct 30 04:20:23.298876 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.298876    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"
Oct 30 04:20:23.299137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.298951    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" event={"ID":"745aa4c9-1934-4912-9757-5e6de52dd2c1","Type":"ContainerDied","Data":"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"}
Oct 30 04:20:23.299137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.298990    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5" event={"ID":"745aa4c9-1934-4912-9757-5e6de52dd2c1","Type":"ContainerDied","Data":"3d3a772893b9ade94964d48226056e94777bd708f754606b091223ef4bf220a5"}
Oct 30 04:20:23.299137 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.299009    4528 scope.go:122] "RemoveContainer" containerID="2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"
Oct 30 04:20:23.310291 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.310183    4528 scope.go:122] "RemoveContainer" containerID="2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"
Oct 30 04:20:23.310483 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:23.310466    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632\": container with ID starting with 2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632 not found: ID does not exist" containerID="2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"
Oct 30 04:20:23.310551 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.310490    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632"} err="failed to get container status \"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632\": rpc error: code = NotFound desc = could not find container \"2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632\": container with ID starting with 2f506a0c88871fea8ad75190cc5feb227ffb646e4a4def98d4c7b17ebf3f8632 not found: ID does not exist"
Oct 30 04:20:23.346950 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.346923    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:23.350659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.350636    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-69cb6bdcdc-952l5"]
Oct 30 04:20:23.892574 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:23.892538    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745aa4c9-1934-4912-9757-5e6de52dd2c1" path="/var/lib/kubelet/pods/745aa4c9-1934-4912-9757-5e6de52dd2c1/volumes"
Oct 30 04:20:24.236575 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:24.236540    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:24.456095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:24.456055    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:26.583930 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.583894    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.611699 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.611672    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.684760 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.684724    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.684953 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.684863    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.698785 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.698759    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.799154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.799119    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.824296 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.824266    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:26.957454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:26.957420    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:27.294573 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:27.294543    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1863677_bc90_4a97_af48_5ee0c0be4ab1.slice/crio-594dd4e4c9d39994f41423724ee45f0f556389703c497d06bda3424f99fa6e08 WatchSource:0}: Error finding container 594dd4e4c9d39994f41423724ee45f0f556389703c497d06bda3424f99fa6e08: Status 404 returned error can't find the container with id 594dd4e4c9d39994f41423724ee45f0f556389703c497d06bda3424f99fa6e08
Oct 30 04:20:27.303909 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:27.303884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:27.317046 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:27.317020    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerStarted","Data":"594dd4e4c9d39994f41423724ee45f0f556389703c497d06bda3424f99fa6e08"}
Oct 30 04:20:28.322442 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:28.322408    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerStarted","Data":"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"}
Oct 30 04:20:31.336799 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:31.336761    4528 generic.go:356] "Generic (PLEG): container finished" podID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c" exitCode=0
Oct 30 04:20:31.337218 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:31.336814    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerDied","Data":"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"}
Oct 30 04:20:32.344536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:32.344439    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerStarted","Data":"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"}
Oct 30 04:20:32.344536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:32.344523    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerStarted","Data":"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"}
Oct 30 04:20:32.345084 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:32.344560    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerStarted","Data":"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"}
Oct 30 04:20:32.419970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:32.419913    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.035318276 podStartE2EDuration="10.419898132s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384579856s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:32.41492716 +0000 UTC m=+5990.639176437" watchObservedRunningTime="2026-10-30 04:20:32.419898132 +0000 UTC m=+5990.644147404"
Oct 30 04:20:34.630888 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:34.630851    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:20:36.958303 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:36.958263    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:37.968791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:37.968754    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:39.111050 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.111018    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:39.669002 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.668970    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:39.755443 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.755408    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:39.755652 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.755624    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="71d761ac-9500-46a5-a472-b3c24baffdd2" containerName="otc-container" containerID="cri-o://b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0" gracePeriod=30
Oct 30 04:20:39.786622 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.786590    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:39.904353 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:39.904321    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:40.019473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:40.019400    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:40.219298 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:40.219263    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:40.999659 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:40.999635    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:41.109547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109458    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\" (UniqueName: \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\") pod \"71d761ac-9500-46a5-a472-b3c24baffdd2\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") "
Oct 30 04:20:41.109547 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109511    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\") pod \"71d761ac-9500-46a5-a472-b3c24baffdd2\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") "
Oct 30 04:20:41.109742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109554    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\") pod \"71d761ac-9500-46a5-a472-b3c24baffdd2\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") "
Oct 30 04:20:41.109742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109585    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\") pod \"71d761ac-9500-46a5-a472-b3c24baffdd2\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") "
Oct 30 04:20:41.109742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109723    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\") pod \"71d761ac-9500-46a5-a472-b3c24baffdd2\" (UID: \"71d761ac-9500-46a5-a472-b3c24baffdd2\") "
Oct 30 04:20:41.110020 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.109997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal" pod "71d761ac-9500-46a5-a472-b3c24baffdd2" (UID: "71d761ac-9500-46a5-a472-b3c24baffdd2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:41.111633 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.111609    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf" pod "71d761ac-9500-46a5-a472-b3c24baffdd2" (UID: "71d761ac-9500-46a5-a472-b3c24baffdd2"). InnerVolumeSpecName "kube-api-access-k62kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:41.111769 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.111743    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs" pod "71d761ac-9500-46a5-a472-b3c24baffdd2" (UID: "71d761ac-9500-46a5-a472-b3c24baffdd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:41.111899 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.111801    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert" pod "71d761ac-9500-46a5-a472-b3c24baffdd2" (UID: "71d761ac-9500-46a5-a472-b3c24baffdd2"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:41.111899 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.111843    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs" pod "71d761ac-9500-46a5-a472-b3c24baffdd2" (UID: "71d761ac-9500-46a5-a472-b3c24baffdd2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:41.211099 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.211066    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:41.211099 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.211092    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62kf\" (UniqueName: \"kubernetes.io/projected/71d761ac-9500-46a5-a472-b3c24baffdd2-kube-api-access-k62kf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:41.211099 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.211103    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:41.211348 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.211116    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71d761ac-9500-46a5-a472-b3c24baffdd2-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:41.211348 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.211125    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/71d761ac-9500-46a5-a472-b3c24baffdd2-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:41.387139 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.387109    4528 generic.go:356] "Generic (PLEG): container finished" podID="71d761ac-9500-46a5-a472-b3c24baffdd2" containerID="b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0" exitCode=0
Oct 30 04:20:41.387556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.387155    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1"
Oct 30 04:20:41.387556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.387180    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"71d761ac-9500-46a5-a472-b3c24baffdd2","Type":"ContainerDied","Data":"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"}
Oct 30 04:20:41.387556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.387221    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"71d761ac-9500-46a5-a472-b3c24baffdd2","Type":"ContainerDied","Data":"ef0b7de14d0c9f46d6d2fea5c06e4a3ca34d2bc0f2121d89d175a0064799e2c0"}
Oct 30 04:20:41.387556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.387243    4528 scope.go:122] "RemoveContainer" containerID="b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"
Oct 30 04:20:41.396111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.396091    4528 scope.go:122] "RemoveContainer" containerID="b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"
Oct 30 04:20:41.396364 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:41.396344    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0\": container with ID starting with b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0 not found: ID does not exist" containerID="b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"
Oct 30 04:20:41.396411 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.396372    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0"} err="failed to get container status \"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0\": rpc error: code = NotFound desc = could not find container \"b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0\": container with ID starting with b17043b18b0bb24f2e55b5ffad1e9c35895e743d80d034c4cb88b3a3401d66c0 not found: ID does not exist"
Oct 30 04:20:41.431197 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.431163    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:41.435806 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.435784    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"]
Oct 30 04:20:41.892811 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.892779    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d761ac-9500-46a5-a472-b3c24baffdd2" path="/var/lib/kubelet/pods/71d761ac-9500-46a5-a472-b3c24baffdd2/volumes"
Oct 30 04:20:41.957592 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.957559    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:41.962739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:41.962717    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:42.027021 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:42.026988    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:42.396418 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:42.396393    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:20:46.357737 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.357692    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358008    4528 cpu_manager.go:414] "containerMap: removing container" podUID="745aa4c9-1934-4912-9757-5e6de52dd2c1" containerName="thanos-querier"
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358024    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745aa4c9-1934-4912-9757-5e6de52dd2c1" containerName="thanos-querier"
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358052    4528 cpu_manager.go:414] "containerMap: removing container" podUID="71d761ac-9500-46a5-a472-b3c24baffdd2" containerName="otc-container"
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358060    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d761ac-9500-46a5-a472-b3c24baffdd2" containerName="otc-container"
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358132    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="745aa4c9-1934-4912-9757-5e6de52dd2c1" containerName="thanos-querier"
Oct 30 04:20:46.358195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.358144    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d761ac-9500-46a5-a472-b3c24baffdd2" containerName="otc-container"
Oct 30 04:20:46.362601 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.362585    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.385751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.385725    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\""
Oct 30 04:20:46.385751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.385734    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:20:46.385976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.385872    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:20:46.385976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.385878    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:20:46.395662 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.395641    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:20:46.397062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.397041    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:46.456707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.456670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.456707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.456708    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhdv\" (UniqueName: \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.456943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.456771    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.456943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.456793    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.456943 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.456896    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558004 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.557968    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.558009    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhdv\" (UniqueName: \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.558031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.558054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558123 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.558103    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.558781 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.558755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.560552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.560527    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.560656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.560589    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.560656 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.560622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.579040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.579013    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhdv\" (UniqueName: \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\") pod \"data-science-collector-collector-0\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.674600 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.674572    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:46.969806 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.969733    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:46.974523 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:20:46.974494    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d9f609_cbde_40ac_9258_b9ce34b278d7.slice/crio-2564be9c4aaa1ccace3f94d22aa41a3a0ee61a24538794cac60c6b8ba567b66c WatchSource:0}: Error finding container 2564be9c4aaa1ccace3f94d22aa41a3a0ee61a24538794cac60c6b8ba567b66c: Status 404 returned error can't find the container with id 2564be9c4aaa1ccace3f94d22aa41a3a0ee61a24538794cac60c6b8ba567b66c
Oct 30 04:20:46.976715 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.976683    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:46.976783 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:46.976743    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:20:47.419884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:47.419843    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b8d9f609-cbde-40ac-9258-b9ce34b278d7","Type":"ContainerStarted","Data":"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"}
Oct 30 04:20:47.420395 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:47.419905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b8d9f609-cbde-40ac-9258-b9ce34b278d7","Type":"ContainerStarted","Data":"2564be9c4aaa1ccace3f94d22aa41a3a0ee61a24538794cac60c6b8ba567b66c"}
Oct 30 04:20:47.457664 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:47.457614    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.457600199 podStartE2EDuration="1.457600199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:47.457287214 +0000 UTC m=+6005.681536491" watchObservedRunningTime="2026-10-30 04:20:47.457600199 +0000 UTC m=+6005.681849476"
Oct 30 04:20:52.500900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:52.500863    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:52.622994 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:52.622951    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:52.743512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:52.743477    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:52.944520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:52.944489    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:54.461160 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:54.461129    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:54.583206 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:54.583159    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:54.698499 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:54.698466    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:57.048322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:57.047887    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:57.048322 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:57.048143    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" containerName="otc-container" containerID="cri-o://b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e" gracePeriod=30
Oct 30 04:20:59.384523 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.384496    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.462492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462404    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\") pod \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") "
Oct 30 04:20:59.462492 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462440    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\") pod \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") "
Oct 30 04:20:59.462764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462665    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\" (UniqueName: \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\") pod \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") "
Oct 30 04:20:59.462764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462736    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\") pod \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") "
Oct 30 04:20:59.462898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462780    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\") pod \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\" (UID: \"b8d9f609-cbde-40ac-9258-b9ce34b278d7\") "
Oct 30 04:20:59.462898 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.462860    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal" pod "b8d9f609-cbde-40ac-9258-b9ce34b278d7" (UID: "b8d9f609-cbde-40ac-9258-b9ce34b278d7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:20:59.463082 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.463058    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b8d9f609-cbde-40ac-9258-b9ce34b278d7-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:59.465716 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.465681    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs" pod "b8d9f609-cbde-40ac-9258-b9ce34b278d7" (UID: "b8d9f609-cbde-40ac-9258-b9ce34b278d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:59.465903 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.465767    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv" pod "b8d9f609-cbde-40ac-9258-b9ce34b278d7" (UID: "b8d9f609-cbde-40ac-9258-b9ce34b278d7"). InnerVolumeSpecName "kube-api-access-7bhdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:59.465986 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.465918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs" pod "b8d9f609-cbde-40ac-9258-b9ce34b278d7" (UID: "b8d9f609-cbde-40ac-9258-b9ce34b278d7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:59.468926 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.468900    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert" pod "b8d9f609-cbde-40ac-9258-b9ce34b278d7" (UID: "b8d9f609-cbde-40ac-9258-b9ce34b278d7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:20:59.474118 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474090    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:20:59.474320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474292    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" podUID="0b432647-4f35-49b6-b102-f65b0fb95836" containerName="thanos-querier" containerID="cri-o://1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55" gracePeriod=30
Oct 30 04:20:59.474391 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474317    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b8d9f609-cbde-40ac-9258-b9ce34b278d7","Type":"ContainerDied","Data":"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"}
Oct 30 04:20:59.474454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474296    4528 generic.go:356] "Generic (PLEG): container finished" podID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" containerID="b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e" exitCode=0
Oct 30 04:20:59.474454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474400    4528 scope.go:122] "RemoveContainer" containerID="b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"
Oct 30 04:20:59.474454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474420    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b8d9f609-cbde-40ac-9258-b9ce34b278d7","Type":"ContainerDied","Data":"2564be9c4aaa1ccace3f94d22aa41a3a0ee61a24538794cac60c6b8ba567b66c"}
Oct 30 04:20:59.474618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.474551    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.490402 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.490304    4528 scope.go:122] "RemoveContainer" containerID="b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"
Oct 30 04:20:59.490683 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:20:59.490659    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e\": container with ID starting with b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e not found: ID does not exist" containerID="b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"
Oct 30 04:20:59.490797 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.490689    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e"} err="failed to get container status \"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e\": rpc error: code = NotFound desc = could not find container \"b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e\": container with ID starting with b4b6072f84e1d68149eaf3003a4e1c35bc732baabb4e6ddf8d5bb7107b6ff59e not found: ID does not exist"
Oct 30 04:20:59.546252 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.546220    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:59.546483 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.546456    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:59.552967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.552914    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:59.564369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.564343    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhdv\" (UniqueName: \"kubernetes.io/projected/b8d9f609-cbde-40ac-9258-b9ce34b278d7-kube-api-access-7bhdv\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:59.564369 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.564370    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:59.564534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.564386    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:59.564534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.564399    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d9f609-cbde-40ac-9258-b9ce34b278d7-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:20:59.595779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.595751    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:59.596235 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.596217    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" containerName="otc-container"
Oct 30 04:20:59.596305 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.596237    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" containerName="otc-container"
Oct 30 04:20:59.596361 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.596307    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" containerName="otc-container"
Oct 30 04:20:59.599462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.599439    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.613184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.612757    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:20:59.613184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.612845    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-r7bt7\""
Oct 30 04:20:59.613184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.612977    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:20:59.613184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.613045    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:20:59.613184 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.613131    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:20:59.623035 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.622960    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:20:59.657352 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.657314    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:20:59.657660 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.657629    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="prometheus" containerID="cri-o://dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5" gracePeriod=600
Oct 30 04:20:59.658661 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.658058    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="thanos-sidecar" containerID="cri-o://3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461" gracePeriod=600
Oct 30 04:20:59.658661 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.658186    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="config-reloader" containerID="cri-o://04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10" gracePeriod=600
Oct 30 04:20:59.665320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.665293    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.665466 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.665348    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.665542 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.665488    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.665602 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.665576    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.665602 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.665606    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnng\" (UniqueName: \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.673045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.673003    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:59.766448 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.766419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.766625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.766489    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.766625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.766521    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnng\" (UniqueName: \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.766625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.766542    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.766625 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.766572    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.767875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.767854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.769592 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.769566    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.769714 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.769693    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.769770 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.769717    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.773426 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.773409    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:20:59.788517 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.788485    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:59.789485 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.789462    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnng\" (UniqueName: \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\") pod \"data-science-collector-collector-0\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.867690 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.867661    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\" (UniqueName: \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\") pod \"0b432647-4f35-49b6-b102-f65b0fb95836\" (UID: \"0b432647-4f35-49b6-b102-f65b0fb95836\") "
Oct 30 04:20:59.869580 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.869556    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz" pod "0b432647-4f35-49b6-b102-f65b0fb95836" (UID: "0b432647-4f35-49b6-b102-f65b0fb95836"). InnerVolumeSpecName "kube-api-access-q6hwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:20:59.892720 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.892684    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d9f609-cbde-40ac-9258-b9ce34b278d7" path="/var/lib/kubelet/pods/b8d9f609-cbde-40ac-9258-b9ce34b278d7/volumes"
Oct 30 04:20:59.907562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.907534    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:20:59.917415 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.917393    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:20:59.969701 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:20:59.969468    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hwz\" (UniqueName: \"kubernetes.io/projected/0b432647-4f35-49b6-b102-f65b0fb95836-kube-api-access-q6hwz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.213167 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.213130    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:00.218273 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:21:00.218230    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded032062_a8b0_444c_a9a3_f2dd55fe7bff.slice/crio-f2757dc2868cac4836990c67a45957f04893bb69ae6ab1a949294b3ffb8b1195 WatchSource:0}: Error finding container f2757dc2868cac4836990c67a45957f04893bb69ae6ab1a949294b3ffb8b1195: Status 404 returned error can't find the container with id f2757dc2868cac4836990c67a45957f04893bb69ae6ab1a949294b3ffb8b1195
Oct 30 04:21:00.220561 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.220532    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:21:00.220626 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.220603    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:21:00.257933 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.257662    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:00.326052 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.325971    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:00.439317 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.439291    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:00.446747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.446719    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:00.480711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480621    4528 generic.go:356] "Generic (PLEG): container finished" podID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461" exitCode=0
Oct 30 04:21:00.480711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480650    4528 generic.go:356] "Generic (PLEG): container finished" podID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10" exitCode=0
Oct 30 04:21:00.480711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480659    4528 generic.go:356] "Generic (PLEG): container finished" podID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5" exitCode=0
Oct 30 04:21:00.480711 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480696    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:00.481013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480719    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerDied","Data":"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"}
Oct 30 04:21:00.481013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerDied","Data":"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"}
Oct 30 04:21:00.481013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480847    4528 scope.go:122] "RemoveContainer" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"
Oct 30 04:21:00.481013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480870    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerDied","Data":"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"}
Oct 30 04:21:00.481013 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.480910    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"c1863677-bc90-4a97-af48-5ee0c0be4ab1","Type":"ContainerDied","Data":"594dd4e4c9d39994f41423724ee45f0f556389703c497d06bda3424f99fa6e08"}
Oct 30 04:21:00.483471 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.483443    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed032062-a8b0-444c-a9a3-f2dd55fe7bff","Type":"ContainerStarted","Data":"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"}
Oct 30 04:21:00.483471 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.483455    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" containerName="otc-container" containerID="cri-o://adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742" gracePeriod=30
Oct 30 04:21:00.483654 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.483485    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed032062-a8b0-444c-a9a3-f2dd55fe7bff","Type":"ContainerStarted","Data":"f2757dc2868cac4836990c67a45957f04893bb69ae6ab1a949294b3ffb8b1195"}
Oct 30 04:21:00.484767 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.484744    4528 generic.go:356] "Generic (PLEG): container finished" podID="0b432647-4f35-49b6-b102-f65b0fb95836" containerID="1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55" exitCode=0
Oct 30 04:21:00.484875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.484784    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" event={"ID":"0b432647-4f35-49b6-b102-f65b0fb95836","Type":"ContainerDied","Data":"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"}
Oct 30 04:21:00.484875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.484795    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"
Oct 30 04:21:00.484875 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.484804    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb" event={"ID":"0b432647-4f35-49b6-b102-f65b0fb95836","Type":"ContainerDied","Data":"0eea3d78bab1da5f8488434d2bee0c6a36ca040e3b24f55e84b31bb80a56ef6f"}
Oct 30 04:21:00.492405 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.492382    4528 scope.go:122] "RemoveContainer" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"
Oct 30 04:21:00.539603 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.539551    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.53953698 podStartE2EDuration="1.53953698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:00.538571695 +0000 UTC m=+6018.762820975" watchObservedRunningTime="2026-10-30 04:21:00.53953698 +0000 UTC m=+6018.763786257"
Oct 30 04:21:00.567462 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.567429    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:00.575340 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575353    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575651    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575712    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575783    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575860    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575896    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575925    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.575963    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.576030    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\") pod \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\" (UID: \"c1863677-bc90-4a97-af48-5ee0c0be4ab1\") "
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.577116    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:00.577965 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.577563    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:00.579023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.578038    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.579023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.578247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.579502 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.579474    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.579802 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.579774    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:00.581295 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.581236    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:00.583155 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.583127    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:21:00.583264 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.583216    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:21:00.583924 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.583793    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "kube-api-access-mz7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:00.583924 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.583845    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.585599 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.585572    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.585900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.585873    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:21:00.590276 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.590253    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-2x5mb"]
Oct 30 04:21:00.599158 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.599037    4528 scope.go:122] "RemoveContainer" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"
Oct 30 04:21:00.599801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.599774    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config" pod "c1863677-bc90-4a97-af48-5ee0c0be4ab1" (UID: "c1863677-bc90-4a97-af48-5ee0c0be4ab1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.608073 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.608014    4528 scope.go:122] "RemoveContainer" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"
Oct 30 04:21:00.617115 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617028    4528 scope.go:122] "RemoveContainer" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"
Oct 30 04:21:00.617386 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:00.617358    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": container with ID starting with 3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461 not found: ID does not exist" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"
Oct 30 04:21:00.617454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617399    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"} err="failed to get container status \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": rpc error: code = NotFound desc = could not find container \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": container with ID starting with 3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461 not found: ID does not exist"
Oct 30 04:21:00.617454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617422    4528 scope.go:122] "RemoveContainer" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"
Oct 30 04:21:00.617688 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:00.617662    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": container with ID starting with 04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10 not found: ID does not exist" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"
Oct 30 04:21:00.617747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617698    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"} err="failed to get container status \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": rpc error: code = NotFound desc = could not find container \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": container with ID starting with 04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10 not found: ID does not exist"
Oct 30 04:21:00.617747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617722    4528 scope.go:122] "RemoveContainer" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"
Oct 30 04:21:00.617979 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:00.617961    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": container with ID starting with dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5 not found: ID does not exist" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"
Oct 30 04:21:00.618060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.617988    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"} err="failed to get container status \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": rpc error: code = NotFound desc = could not find container \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": container with ID starting with dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5 not found: ID does not exist"
Oct 30 04:21:00.618060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618010    4528 scope.go:122] "RemoveContainer" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"
Oct 30 04:21:00.618278 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:00.618250    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": container with ID starting with 688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c not found: ID does not exist" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"
Oct 30 04:21:00.618342 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618286    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"} err="failed to get container status \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": rpc error: code = NotFound desc = could not find container \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": container with ID starting with 688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c not found: ID does not exist"
Oct 30 04:21:00.618342 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618307    4528 scope.go:122] "RemoveContainer" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"
Oct 30 04:21:00.618596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618575    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"} err="failed to get container status \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": rpc error: code = NotFound desc = could not find container \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": container with ID starting with 3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461 not found: ID does not exist"
Oct 30 04:21:00.618596 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618596    4528 scope.go:122] "RemoveContainer" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"
Oct 30 04:21:00.618893 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618871    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"} err="failed to get container status \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": rpc error: code = NotFound desc = could not find container \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": container with ID starting with 04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10 not found: ID does not exist"
Oct 30 04:21:00.618979 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.618894    4528 scope.go:122] "RemoveContainer" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"
Oct 30 04:21:00.619159 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619141    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"} err="failed to get container status \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": rpc error: code = NotFound desc = could not find container \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": container with ID starting with dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5 not found: ID does not exist"
Oct 30 04:21:00.619222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619159    4528 scope.go:122] "RemoveContainer" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"
Oct 30 04:21:00.619435 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619398    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"} err="failed to get container status \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": rpc error: code = NotFound desc = could not find container \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": container with ID starting with 688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c not found: ID does not exist"
Oct 30 04:21:00.619512 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619435    4528 scope.go:122] "RemoveContainer" containerID="3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"
Oct 30 04:21:00.619754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619729    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461"} err="failed to get container status \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": rpc error: code = NotFound desc = could not find container \"3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461\": container with ID starting with 3dde7b89c6dab9638a879a0e15cbf64f5c66d4fe1b6571e36750dc06ced62461 not found: ID does not exist"
Oct 30 04:21:00.619754 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.619754    4528 scope.go:122] "RemoveContainer" containerID="04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"
Oct 30 04:21:00.620067 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620045    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10"} err="failed to get container status \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": rpc error: code = NotFound desc = could not find container \"04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10\": container with ID starting with 04fa43a15a1178332c5dedbf00767dea9e5a10f9518256bd39155d1dfbff0d10 not found: ID does not exist"
Oct 30 04:21:00.620182 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620068    4528 scope.go:122] "RemoveContainer" containerID="dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"
Oct 30 04:21:00.620337 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620312    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5"} err="failed to get container status \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": rpc error: code = NotFound desc = could not find container \"dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5\": container with ID starting with dbfec3a8cbe7a7097f2b426ffaf56d22e025a45ff2d292ccdf486b612ee472f5 not found: ID does not exist"
Oct 30 04:21:00.620436 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620338    4528 scope.go:122] "RemoveContainer" containerID="688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"
Oct 30 04:21:00.620671 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620650    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c"} err="failed to get container status \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": rpc error: code = NotFound desc = could not find container \"688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c\": container with ID starting with 688b31ad8da6b0d711735932ea48409e31941c5965187ffb92800b962a1fae5c not found: ID does not exist"
Oct 30 04:21:00.620740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.620671    4528 scope.go:122] "RemoveContainer" containerID="1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"
Oct 30 04:21:00.628484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.628460    4528 scope.go:122] "RemoveContainer" containerID="1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"
Oct 30 04:21:00.628784 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:00.628760    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55\": container with ID starting with 1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55 not found: ID does not exist" containerID="1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"
Oct 30 04:21:00.628885 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.628794    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55"} err="failed to get container status \"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55\": rpc error: code = NotFound desc = could not find container \"1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55\": container with ID starting with 1d630fce8fa8245d1767e1873154aa54300a143f8c66c7432cec91f09fd04b55 not found: ID does not exist"
Oct 30 04:21:00.677333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677301    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677338    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677354    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677368    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677382    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677420    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677433    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677444    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1863677-bc90-4a97-af48-5ee0c0be4ab1-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677456    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677472    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677487 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677484    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1863677-bc90-4a97-af48-5ee0c0be4ab1-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677496    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz7xz\" (UniqueName: \"kubernetes.io/projected/c1863677-bc90-4a97-af48-5ee0c0be4ab1-kube-api-access-mz7xz\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.677984 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.677509    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1863677-bc90-4a97-af48-5ee0c0be4ab1-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.689957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.689937    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:21:00.722366 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.722340    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:00.778880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.778733    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:21:00.823226 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.823194    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:00.829307 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.829278    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:00.879534 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879499    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\") pod \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") "
Oct 30 04:21:00.879703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879546    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\" (UniqueName: \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\") pod \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") "
Oct 30 04:21:00.879703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879581    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\") pod \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") "
Oct 30 04:21:00.879703 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879634    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\") pod \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") "
Oct 30 04:21:00.879937 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879734    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\") pod \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\" (UID: \"ed032062-a8b0-444c-a9a3-f2dd55fe7bff\") "
Oct 30 04:21:00.879937 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879883    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal" pod "ed032062-a8b0-444c-a9a3-f2dd55fe7bff" (UID: "ed032062-a8b0-444c-a9a3-f2dd55fe7bff"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:00.880049 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.879972    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.881751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.881720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs" pod "ed032062-a8b0-444c-a9a3-f2dd55fe7bff" (UID: "ed032062-a8b0-444c-a9a3-f2dd55fe7bff"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.881887 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.881785    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert" pod "ed032062-a8b0-444c-a9a3-f2dd55fe7bff" (UID: "ed032062-a8b0-444c-a9a3-f2dd55fe7bff"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.881887 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.881850    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng" pod "ed032062-a8b0-444c-a9a3-f2dd55fe7bff" (UID: "ed032062-a8b0-444c-a9a3-f2dd55fe7bff"). InnerVolumeSpecName "kube-api-access-cdnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:00.881887 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.881852    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs" pod "ed032062-a8b0-444c-a9a3-f2dd55fe7bff" (UID: "ed032062-a8b0-444c-a9a3-f2dd55fe7bff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:00.980672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.980637    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.980672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.980665    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdnng\" (UniqueName: \"kubernetes.io/projected/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-kube-api-access-cdnng\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.980672 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.980677    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:00.980930 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:00.980688    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed032062-a8b0-444c-a9a3-f2dd55fe7bff-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:01.305553 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.305521    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:01.435969 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.435930    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:01.491617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.491534    4528 generic.go:356] "Generic (PLEG): container finished" podID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" containerID="adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742" exitCode=2
Oct 30 04:21:01.491617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.491586    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:01.491617 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.491605    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed032062-a8b0-444c-a9a3-f2dd55fe7bff","Type":"ContainerDied","Data":"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"}
Oct 30 04:21:01.492133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.491645    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed032062-a8b0-444c-a9a3-f2dd55fe7bff","Type":"ContainerDied","Data":"f2757dc2868cac4836990c67a45957f04893bb69ae6ab1a949294b3ffb8b1195"}
Oct 30 04:21:01.492133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.491668    4528 scope.go:122] "RemoveContainer" containerID="adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"
Oct 30 04:21:01.500195 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.500179    4528 scope.go:122] "RemoveContainer" containerID="adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"
Oct 30 04:21:01.500483 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:01.500449    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742\": container with ID starting with adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742 not found: ID does not exist" containerID="adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"
Oct 30 04:21:01.500587 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.500492    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742"} err="failed to get container status \"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742\": rpc error: code = NotFound desc = could not find container \"adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742\": container with ID starting with adfb441e27a2b29bb5c91537c252a707ab3ffa3b3ab1c45682b1a7e5e2edf742 not found: ID does not exist"
Oct 30 04:21:01.535537 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.535499    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:01.539525 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.539495    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:01.557060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.557033    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:01.891915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.891884    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b432647-4f35-49b6-b102-f65b0fb95836" path="/var/lib/kubelet/pods/0b432647-4f35-49b6-b102-f65b0fb95836/volumes"
Oct 30 04:21:01.892327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.892314    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" path="/var/lib/kubelet/pods/c1863677-bc90-4a97-af48-5ee0c0be4ab1/volumes"
Oct 30 04:21:01.892780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:01.892768    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" path="/var/lib/kubelet/pods/ed032062-a8b0-444c-a9a3-f2dd55fe7bff/volumes"
Oct 30 04:21:04.632981 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:04.632946    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:21:11.668095 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:11.668053    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:11.796350 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:11.796307    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:11.911007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:11.910964    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:12.032555 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:12.032462    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:12.222804 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:12.222769    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:12.648060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:12.648028    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:12.818971 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:12.818935    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:13.013728 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.013622    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:13.543690 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.543653    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:13.544097 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544080    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="prometheus"
Oct 30 04:21:13.544163 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544102    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="prometheus"
Oct 30 04:21:13.544163 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544123    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="config-reloader"
Oct 30 04:21:13.544163 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544131    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="config-reloader"
Oct 30 04:21:13.544163 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544142    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" containerName="otc-container"
Oct 30 04:21:13.544163 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544150    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" containerName="otc-container"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544166    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0b432647-4f35-49b6-b102-f65b0fb95836" containerName="thanos-querier"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544174    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b432647-4f35-49b6-b102-f65b0fb95836" containerName="thanos-querier"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544188    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="thanos-sidecar"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544197    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="thanos-sidecar"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544208    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="init-config-reloader"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544216    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="init-config-reloader"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544303    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="prometheus"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544314    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="config-reloader"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544324    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed032062-a8b0-444c-a9a3-f2dd55fe7bff" containerName="otc-container"
Oct 30 04:21:13.544333 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544335    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b432647-4f35-49b6-b102-f65b0fb95836" containerName="thanos-querier"
Oct 30 04:21:13.544693 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.544345    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1863677-bc90-4a97-af48-5ee0c0be4ab1" containerName="thanos-sidecar"
Oct 30 04:21:13.547609 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.547589    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.562351 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.562325    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 04:21:13.572281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.572252    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\""
Oct 30 04:21:13.572281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.572268    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\""
Oct 30 04:21:13.572484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.572328    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kfs49\""
Oct 30 04:21:13.572484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.572402    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\""
Oct 30 04:21:13.573076 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.573037    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-1232cc21\""
Oct 30 04:21:13.573221 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.573202    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\""
Oct 30 04:21:13.574586 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.574563    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:13.585739 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.585705    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:13.685045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.684996    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.685223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.685154    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.685299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.685252    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.685299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.685287    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.685299 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.685313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.710981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.710934    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:13.786078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786045    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.786312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786091    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.786312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786163    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.786312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786195    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.786312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.786896 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.786871    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.788636 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.788616    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.788748 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.788647    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.788748 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.788658    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.807094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.807034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\") pod \"data-science-collector-collector-0\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") " pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:13.825335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.825306    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:13.862452 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:13.862416    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:14.181343 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.181308    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:14.185915 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:21:14.185886    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb394e481_f889_4a02_997b_c29e2278c32e.slice/crio-60b5f1a3afe64ece78ac84c0502df961fbecc379091d896efe8b98174ef1c70e WatchSource:0}: Error finding container 60b5f1a3afe64ece78ac84c0502df961fbecc379091d896efe8b98174ef1c70e: Status 404 returned error can't find the container with id 60b5f1a3afe64ece78ac84c0502df961fbecc379091d896efe8b98174ef1c70e
Oct 30 04:21:14.187981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.187949    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:21:14.188051 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.188011    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 30 04:21:14.550183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.550085    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b394e481-f889-4a02-997b-c29e2278c32e","Type":"ContainerStarted","Data":"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"}
Oct 30 04:21:14.550183 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.550145    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b394e481-f889-4a02-997b-c29e2278c32e","Type":"ContainerStarted","Data":"60b5f1a3afe64ece78ac84c0502df961fbecc379091d896efe8b98174ef1c70e"}
Oct 30 04:21:14.592413 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:14.589478    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.5894512440000002 podStartE2EDuration="1.589451244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:14.586654571 +0000 UTC m=+6032.810903851" watchObservedRunningTime="2026-10-30 04:21:14.589451244 +0000 UTC m=+6032.813700522"
Oct 30 04:21:16.421482 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.421447    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:16.424782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.424760    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:16.437170 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.437145    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-xb4sw\""
Oct 30 04:21:16.448154 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.448097    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:16.608612 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.608577    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvqkw\" (UniqueName: \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp\" (UID: \"191dfbf2-08c7-4ed9-89e0-76f33931c2a6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:16.630531 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.630492    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:16.709275 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.709199    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvqkw\" (UniqueName: \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp\" (UID: \"191dfbf2-08c7-4ed9-89e0-76f33931c2a6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:16.730520 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.730497    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvqkw\" (UniqueName: \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\") pod \"thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp\" (UID: \"191dfbf2-08c7-4ed9-89e0-76f33931c2a6\") " pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:16.736228 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.736201    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:16.888208 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:16.888167    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:17.002300 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.002214    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:17.028734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.028702    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:17.033685 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:21:17.033650    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod191dfbf2_08c7_4ed9_89e0_76f33931c2a6.slice/crio-6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4 WatchSource:0}: Error finding container 6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4: Status 404 returned error can't find the container with id 6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4
Oct 30 04:21:17.279111 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.279014    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:17.345454 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.345408    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:17.349271 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.349238    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:17.349469 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.349401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.353804 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.353780    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.367345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.367321    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\""
Oct 30 04:21:17.374978 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.374951    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:17.382124 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.381724    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:17.407090 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.407059    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432559    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432559    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432651    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-rgp4j\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432656    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432689    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432719    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432749    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\""
Oct 30 04:21:17.432749 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432758    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\""
Oct 30 04:21:17.433755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.432982    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\""
Oct 30 04:21:17.433755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.433021    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\""
Oct 30 04:21:17.433755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.433048    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\""
Oct 30 04:21:17.433755 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.433063    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-7276v\""
Oct 30 04:21:17.467478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.467440    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\""
Oct 30 04:21:17.467478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.467473    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\""
Oct 30 04:21:17.515847 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.515786    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.515999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.515855    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.515999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.515937    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.515999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.515965    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.515999 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.515986    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516030    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516090    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.516225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.516225 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516215    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9gf\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.516434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516321    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdhd\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516457    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516541    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.516590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.516590 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516581    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.516747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.516643    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.563880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.563786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" event={"ID":"191dfbf2-08c7-4ed9-89e0-76f33931c2a6","Type":"ContainerStarted","Data":"dba702f1f2add02a6cf3b3253aa378a51842ffa4e962ec369e83999e9b485dc1"}
Oct 30 04:21:17.563880 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.563861    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" event={"ID":"191dfbf2-08c7-4ed9-89e0-76f33931c2a6","Type":"ContainerStarted","Data":"6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4"}
Oct 30 04:21:17.597536 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.597484    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" podStartSLOduration=1.59746898 podStartE2EDuration="1.59746898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:17.597191673 +0000 UTC m=+6035.821440947" watchObservedRunningTime="2026-10-30 04:21:17.59746898 +0000 UTC m=+6035.821718257"
Oct 30 04:21:17.617108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617078    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617108 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617112    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617132    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617188    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617278    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617300    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617335 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617395    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9gf\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617453    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdhd\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617480    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617518    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.617915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.617610    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.618403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.618014    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.618403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.618390    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.618961 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.618583    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.618961 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.618638    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.620696    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.621704    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.622745    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.622921    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.623026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.623856 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.623430    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.624267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.623879    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.624267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.623950    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.624267 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.624007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.624432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.624326    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.624432 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.624403    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.624545 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.624487    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.625062 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.625038    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.638809 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.638776    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9gf\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.642508 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.642482    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdhd\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:17.668363 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.668331    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:17.984297 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:17.984256    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:17.989487 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:21:17.989455    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146fa115_70dd_4803_a27e_a8afda6b3bc7.slice/crio-615aff531173044f5133e2b2a2afd073c8a38072c9f768129ae33682aebbabb8 WatchSource:0}: Error finding container 615aff531173044f5133e2b2a2afd073c8a38072c9f768129ae33682aebbabb8: Status 404 returned error can't find the container with id 615aff531173044f5133e2b2a2afd073c8a38072c9f768129ae33682aebbabb8
Oct 30 04:21:18.569979 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:18.569934    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerStarted","Data":"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"}
Oct 30 04:21:18.570473 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:18.570010    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerStarted","Data":"615aff531173044f5133e2b2a2afd073c8a38072c9f768129ae33682aebbabb8"}
Oct 30 04:21:19.838079 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:19.838039    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:19.862259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:19.862225    4528 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:19.938668 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:19.938629    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:19.938870 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:19.938762    4528 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:19.950757 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:19.950732    4528 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:20.048403 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:20.048372    4528 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7bd02a6ced0e86eb5067dae735620268cfbf25efbbfc3866c7163efbdf26dd99/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:20.072836 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:20.072797    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:20.377742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:20.377699    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:20.711231 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:21:20.711196    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4687d6d6_88a9_474b_bcfe_8182f4282a4f.slice/crio-fa4c63befe5adfc9d68f8d298311684cc8ca0c3e52265ce867ad7bc4240fa3da WatchSource:0}: Error finding container fa4c63befe5adfc9d68f8d298311684cc8ca0c3e52265ce867ad7bc4240fa3da: Status 404 returned error can't find the container with id fa4c63befe5adfc9d68f8d298311684cc8ca0c3e52265ce867ad7bc4240fa3da
Oct 30 04:21:20.716885 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:20.716858    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:21.585764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:21.585679    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerStarted","Data":"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"}
Oct 30 04:21:21.585764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:21.585742    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerStarted","Data":"fa4c63befe5adfc9d68f8d298311684cc8ca0c3e52265ce867ad7bc4240fa3da"}
Oct 30 04:21:22.592186 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:22.592152    4528 generic.go:356] "Generic (PLEG): container finished" podID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerID="1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18" exitCode=0
Oct 30 04:21:22.592603 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:22.592228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerDied","Data":"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"}
Oct 30 04:21:22.593719 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:22.593702    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:21:24.602764 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:24.602685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerStarted","Data":"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"}
Oct 30 04:21:25.496980 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.496942    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:25.608900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.608799    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerStarted","Data":"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"}
Oct 30 04:21:25.609312 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.608912    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:25.610509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.610489    4528 generic.go:356] "Generic (PLEG): container finished" podID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f" exitCode=0
Oct 30 04:21:25.610612 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.610524    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerDied","Data":"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"}
Oct 30 04:21:25.671440 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:25.671394    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.47368657 podStartE2EDuration="8.671379999s" totalImagesPullingTime="1.71199003s" totalInitContainerRuntime="3.485703399s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:25.660496826 +0000 UTC m=+6043.884746128" watchObservedRunningTime="2026-10-30 04:21:25.671379999 +0000 UTC m=+6043.895629276"
Oct 30 04:21:26.620152 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:26.620114    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerStarted","Data":"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"}
Oct 30 04:21:26.620667 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:26.620181    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerStarted","Data":"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"}
Oct 30 04:21:26.620667 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:26.620205    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerStarted","Data":"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"}
Oct 30 04:21:26.677045 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:26.676996    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=6.191082113 podStartE2EDuration="9.676977196s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485895083s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:26.667075696 +0000 UTC m=+6044.891324970" watchObservedRunningTime="2026-10-30 04:21:26.676977196 +0000 UTC m=+6044.901226476"
Oct 30 04:21:30.378465 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:30.378427    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:31.621813 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:31.621782    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:34.631759 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:34.631719    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:21:35.377900 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:35.377862    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:35.383501 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:35.383478    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:35.663007 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:35.662981    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:40.450789 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.450758    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:40.451277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.450971    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" podUID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" containerName="thanos-querier" containerID="cri-o://dba702f1f2add02a6cf3b3253aa378a51842ffa4e962ec369e83999e9b485dc1" gracePeriod=30
Oct 30 04:21:40.515740 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.515705    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:40.631467 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.631398    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:40.686311 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.686278    4528 generic.go:356] "Generic (PLEG): container finished" podID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" containerID="dba702f1f2add02a6cf3b3253aa378a51842ffa4e962ec369e83999e9b485dc1" exitCode=0
Oct 30 04:21:40.686434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.686347    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" event={"ID":"191dfbf2-08c7-4ed9-89e0-76f33931c2a6","Type":"ContainerDied","Data":"dba702f1f2add02a6cf3b3253aa378a51842ffa4e962ec369e83999e9b485dc1"}
Oct 30 04:21:40.686434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.686388    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp" event={"ID":"191dfbf2-08c7-4ed9-89e0-76f33931c2a6","Type":"ContainerDied","Data":"6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4"}
Oct 30 04:21:40.686434 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.686403    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae55e9cc66ced7f682f91b7496f028e35fb38bbc7615fbd9f47503a67af97b4"
Oct 30 04:21:40.691282 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.691263    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:40.725130 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.725051    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\" (UniqueName: \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\") pod \"191dfbf2-08c7-4ed9-89e0-76f33931c2a6\" (UID: \"191dfbf2-08c7-4ed9-89e0-76f33931c2a6\") "
Oct 30 04:21:40.727456 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.727423    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw" pod "191dfbf2-08c7-4ed9-89e0-76f33931c2a6" (UID: "191dfbf2-08c7-4ed9-89e0-76f33931c2a6"). InnerVolumeSpecName "kube-api-access-hvqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:40.825789 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.825755    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvqkw\" (UniqueName: \"kubernetes.io/projected/191dfbf2-08c7-4ed9-89e0-76f33931c2a6-kube-api-access-hvqkw\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:40.918932 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.918899    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:40.919283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.919193    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager" containerID="cri-o://0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc" gracePeriod=120
Oct 30 04:21:40.919283 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.919245    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="config-reloader" containerID="cri-o://35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7" gracePeriod=120
Oct 30 04:21:40.922222 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.922193    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:40.929776 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.929749    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:40.931109 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.931079    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="prometheus" containerID="cri-o://9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e" gracePeriod=600
Oct 30 04:21:40.931323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.931126    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="config-reloader" containerID="cri-o://6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd" gracePeriod=600
Oct 30 04:21:40.931323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:40.931133    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="thanos-sidecar" containerID="cri-o://efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3" gracePeriod=600
Oct 30 04:21:41.053345 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.053315    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:41.173209 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.173180    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:41.236286 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.236253    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:41.236478 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.236457    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="b394e481-f889-4a02-997b-c29e2278c32e" containerName="otc-container" containerID="cri-o://32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f" gracePeriod=30
Oct 30 04:21:41.291923 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.291614    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:41.404179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.404137    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:41.488697 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.488672    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:41.520506 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.520474    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:41.533370 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\") pod \"b394e481-f889-4a02-997b-c29e2278c32e\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") "
Oct 30 04:21:41.533505 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533420    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\") pod \"b394e481-f889-4a02-997b-c29e2278c32e\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") "
Oct 30 04:21:41.533505 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\") pod \"b394e481-f889-4a02-997b-c29e2278c32e\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") "
Oct 30 04:21:41.533505 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533488    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\") pod \"b394e481-f889-4a02-997b-c29e2278c32e\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") "
Oct 30 04:21:41.533708 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533552    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\") pod \"b394e481-f889-4a02-997b-c29e2278c32e\" (UID: \"b394e481-f889-4a02-997b-c29e2278c32e\") "
Oct 30 04:21:41.533976 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.533950    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal" pod "b394e481-f889-4a02-997b-c29e2278c32e" (UID: "b394e481-f889-4a02-997b-c29e2278c32e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:41.536327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.536141    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs" pod "b394e481-f889-4a02-997b-c29e2278c32e" (UID: "b394e481-f889-4a02-997b-c29e2278c32e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.536327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.536292    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x" pod "b394e481-f889-4a02-997b-c29e2278c32e" (UID: "b394e481-f889-4a02-997b-c29e2278c32e"). InnerVolumeSpecName "kube-api-access-lmp9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:41.536541 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.536521    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs" pod "b394e481-f889-4a02-997b-c29e2278c32e" (UID: "b394e481-f889-4a02-997b-c29e2278c32e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.537129 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.537100    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert" pod "b394e481-f889-4a02-997b-c29e2278c32e" (UID: "b394e481-f889-4a02-997b-c29e2278c32e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.621549 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.621511    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.146:9093/-/ready\": dial tcp 10.133.0.146:9093: connect: connection refused"
Oct 30 04:21:41.634955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.634925    4528 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-data-science-collector-ta-client-cert\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.634955 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.634953    4528 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b394e481-f889-4a02-997b-c29e2278c32e-otc-internal\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.635133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.634965    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.635133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.634973    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmp9x\" (UniqueName: \"kubernetes.io/projected/b394e481-f889-4a02-997b-c29e2278c32e-kube-api-access-lmp9x\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.635133 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.634983    4528 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b394e481-f889-4a02-997b-c29e2278c32e-monitoring-tls-certs\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.662024 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.661999    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:41.693179 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.693146    4528 generic.go:356] "Generic (PLEG): container finished" podID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerID="35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7" exitCode=0
Oct 30 04:21:41.693362 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.693222    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerDied","Data":"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"}
Oct 30 04:21:41.695375 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695357    4528 generic.go:356] "Generic (PLEG): container finished" podID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3" exitCode=0
Oct 30 04:21:41.695375 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695371    4528 generic.go:356] "Generic (PLEG): container finished" podID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd" exitCode=0
Oct 30 04:21:41.695375 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695378    4528 generic.go:356] "Generic (PLEG): container finished" podID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e" exitCode=0
Oct 30 04:21:41.695634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695435    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"
Oct 30 04:21:41.695634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695433    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerDied","Data":"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"}
Oct 30 04:21:41.695634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerDied","Data":"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"}
Oct 30 04:21:41.695634 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695620    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerDied","Data":"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"}
Oct 30 04:21:41.695855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695633    4528 scope.go:122] "RemoveContainer" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"
Oct 30 04:21:41.695855 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.695656    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"4687d6d6-88a9-474b-bcfe-8182f4282a4f","Type":"ContainerDied","Data":"fa4c63befe5adfc9d68f8d298311684cc8ca0c3e52265ce867ad7bc4240fa3da"}
Oct 30 04:21:41.696742 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.696725    4528 generic.go:356] "Generic (PLEG): container finished" podID="b394e481-f889-4a02-997b-c29e2278c32e" containerID="32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f" exitCode=0
Oct 30 04:21:41.696808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.696774    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0"
Oct 30 04:21:41.696808 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.696789    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"
Oct 30 04:21:41.696915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.696808    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b394e481-f889-4a02-997b-c29e2278c32e","Type":"ContainerDied","Data":"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"}
Oct 30 04:21:41.696915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.696856    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"b394e481-f889-4a02-997b-c29e2278c32e","Type":"ContainerDied","Data":"60b5f1a3afe64ece78ac84c0502df961fbecc379091d896efe8b98174ef1c70e"}
Oct 30 04:21:41.703874 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.703858    4528 scope.go:122] "RemoveContainer" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"
Oct 30 04:21:41.713264 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.713235    4528 scope.go:122] "RemoveContainer" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"
Oct 30 04:21:41.723579 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.723558    4528 scope.go:122] "RemoveContainer" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"
Oct 30 04:21:41.732019 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.731996    4528 scope.go:122] "RemoveContainer" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"
Oct 30 04:21:41.732368 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:41.732342    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": container with ID starting with efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3 not found: ID does not exist" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"
Oct 30 04:21:41.732460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732385    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"} err="failed to get container status \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": rpc error: code = NotFound desc = could not find container \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": container with ID starting with efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3 not found: ID does not exist"
Oct 30 04:21:41.732460 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732409    4528 scope.go:122] "RemoveContainer" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"
Oct 30 04:21:41.732684 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:41.732662    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": container with ID starting with 6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd not found: ID does not exist" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"
Oct 30 04:21:41.732747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732690    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"} err="failed to get container status \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": rpc error: code = NotFound desc = could not find container \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": container with ID starting with 6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd not found: ID does not exist"
Oct 30 04:21:41.732747 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732707    4528 scope.go:122] "RemoveContainer" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"
Oct 30 04:21:41.732958 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:41.732935    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": container with ID starting with 9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e not found: ID does not exist" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"
Oct 30 04:21:41.733020 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732967    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"} err="failed to get container status \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": rpc error: code = NotFound desc = could not find container \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": container with ID starting with 9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e not found: ID does not exist"
Oct 30 04:21:41.733020 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.732989    4528 scope.go:122] "RemoveContainer" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"
Oct 30 04:21:41.733215 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:41.733194    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": container with ID starting with 0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f not found: ID does not exist" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"
Oct 30 04:21:41.733281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733233    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"} err="failed to get container status \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": rpc error: code = NotFound desc = could not find container \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": container with ID starting with 0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f not found: ID does not exist"
Oct 30 04:21:41.733281 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733252    4528 scope.go:122] "RemoveContainer" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"
Oct 30 04:21:41.733498 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733478    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"} err="failed to get container status \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": rpc error: code = NotFound desc = could not find container \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": container with ID starting with efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3 not found: ID does not exist"
Oct 30 04:21:41.733557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733498    4528 scope.go:122] "RemoveContainer" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"
Oct 30 04:21:41.733751 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733731    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"} err="failed to get container status \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": rpc error: code = NotFound desc = could not find container \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": container with ID starting with 6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd not found: ID does not exist"
Oct 30 04:21:41.733844 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733751    4528 scope.go:122] "RemoveContainer" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"
Oct 30 04:21:41.733985 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733964    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"} err="failed to get container status \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": rpc error: code = NotFound desc = could not find container \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": container with ID starting with 9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e not found: ID does not exist"
Oct 30 04:21:41.734052 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.733987    4528 scope.go:122] "RemoveContainer" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"
Oct 30 04:21:41.734266 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734226    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"} err="failed to get container status \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": rpc error: code = NotFound desc = could not find container \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": container with ID starting with 0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f not found: ID does not exist"
Oct 30 04:21:41.734341 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734268    4528 scope.go:122] "RemoveContainer" containerID="efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"
Oct 30 04:21:41.734524 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734507    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3"} err="failed to get container status \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": rpc error: code = NotFound desc = could not find container \"efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3\": container with ID starting with efb6488d6be8e5fa092c18e730bd7f6270ecf071450197752017007bcad218f3 not found: ID does not exist"
Oct 30 04:21:41.734594 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734524    4528 scope.go:122] "RemoveContainer" containerID="6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"
Oct 30 04:21:41.734738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734722    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd"} err="failed to get container status \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": rpc error: code = NotFound desc = could not find container \"6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd\": container with ID starting with 6c2c286f445dc464f6d5fc1c8ff819652dbacc5dd3d30f190ed57b51ec4fdffd not found: ID does not exist"
Oct 30 04:21:41.734801 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734737    4528 scope.go:122] "RemoveContainer" containerID="9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"
Oct 30 04:21:41.734970 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734949    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e"} err="failed to get container status \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": rpc error: code = NotFound desc = could not find container \"9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e\": container with ID starting with 9c24ae848f72be3ee3f41cb1ef73db45ef4283073faafc02739353a0e170203e not found: ID does not exist"
Oct 30 04:21:41.735040 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.734971    4528 scope.go:122] "RemoveContainer" containerID="0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"
Oct 30 04:21:41.735189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735153    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735194    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735194    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f"} err="failed to get container status \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": rpc error: code = NotFound desc = could not find container \"0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f\": container with ID starting with 0acc232a8760e8fea3fb2530da456ef2b67d960fb8a31bcddffa505165c0ea8f not found: ID does not exist"
Oct 30 04:21:41.735259 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735245    4528 scope.go:122] "RemoveContainer" containerID="32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"
Oct 30 04:21:41.735416 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735225    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735471 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735438    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735533 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735488    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735533 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735520    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735543    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735580    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735604    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735643 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735638    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735664    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735686    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\") pod \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\" (UID: \"4687d6d6-88a9-474b-bcfe-8182f4282a4f\") "
Oct 30 04:21:41.735884 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.735517    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:41.736780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.736005    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.736780 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.736683    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:41.737509 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.737481    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:21:41.738744 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.738707    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.738854 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.738794    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.739173 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.739146    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:41.739997 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.739956    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.740101 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.739988    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.740491 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.740470    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.741189 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.741165    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:21:41.741897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.741250    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "kube-api-access-qkdhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:41.741897 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.741868    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:41.746323 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.746276    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-querier-data-science-thanos-querier-74b6964bcd-dznrp"]
Oct 30 04:21:41.747052 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.747020    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:21:41.748211 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.748168    4528 scope.go:122] "RemoveContainer" containerID="32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"
Oct 30 04:21:41.748547 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:41.748509    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f\": container with ID starting with 32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f not found: ID does not exist" containerID="32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"
Oct 30 04:21:41.748638 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.748559    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f"} err="failed to get container status \"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f\": rpc error: code = NotFound desc = could not find container \"32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f\": container with ID starting with 32aeeeae6e9be8f7f8abf1d5921df8c4747ec5e9ab51f16928ce701639b2587f not found: ID does not exist"
Oct 30 04:21:41.752165 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.752142    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config" pod "4687d6d6-88a9-474b-bcfe-8182f4282a4f" (UID: "4687d6d6-88a9-474b-bcfe-8182f4282a4f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:41.788783 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.788756    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:41.792793 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.792729    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"]
Oct 30 04:21:41.836889 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836857    4528 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.836889 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836886    4528 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-secret-prometheus-web-tls-ca\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836900    4528 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836914    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836928    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836944    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdhd\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-kube-api-access-qkdhd\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836957    4528 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4687d6d6-88a9-474b-bcfe-8182f4282a4f-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836970    4528 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836984    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4687d6d6-88a9-474b-bcfe-8182f4282a4f-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.836995    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4687d6d6-88a9-474b-bcfe-8182f4282a4f-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.837006    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4687d6d6-88a9-474b-bcfe-8182f4282a4f-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:41.837078 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.837044    4528 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" "
Oct 30 04:21:41.848474 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.848454    4528 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085") on node "ip-10-0-134-164.ec2.internal"
Oct 30 04:21:41.891972 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.891939    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" path="/var/lib/kubelet/pods/191dfbf2-08c7-4ed9-89e0-76f33931c2a6/volumes"
Oct 30 04:21:41.892396 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.892376    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b394e481-f889-4a02-997b-c29e2278c32e" path="/var/lib/kubelet/pods/b394e481-f889-4a02-997b-c29e2278c32e/volumes"
Oct 30 04:21:41.937671 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:41.937637    4528 reconciler_common.go:299] "Volume detached for volume \"pvc-e301c60d-9f23-47a7-ac5a-1dbdd3dc1507\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-087f27df3b9790085\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"csi-4fb4ec1f7a44b08271854fe6605474acaf91971f64325d09a68fa5d815443be1\""
Oct 30 04:21:42.033030 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.033000    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:42.039356 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.039325    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"]
Oct 30 04:21:42.181085 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.181050    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:42.312493 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.312455    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:42.427404 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.427369    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:42.541328 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:42.541294    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:43.891915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:43.891884    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" path="/var/lib/kubelet/pods/4687d6d6-88a9-474b-bcfe-8182f4282a4f/volumes"
Oct 30 04:21:44.117319 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:44.117284    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:44.235633 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:44.235543    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:44.407611 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:44.407572    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:46.621008 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:46.620970    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.146:9093/-/ready\": dial tcp 10.133.0.146:9093: connect: connection refused"
Oct 30 04:21:50.052277 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:50.052238    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:50.167272 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:50.167240    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:51.166871 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.166845    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:51.213190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213119    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213154    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213190 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213175    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213389 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213199    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213438 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213401    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213496 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213444    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\") pod \"146fa115-70dd-4803-a27e-a8afda6b3bc7\" (UID: \"146fa115-70dd-4803-a27e-a8afda6b3bc7\") "
Oct 30 04:21:51.213562 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213528    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:21:51.213733 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.213709    4528 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.215557 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.215520    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:21:51.215775 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.215744    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:51.215915 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.215895    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "kube-api-access-lt9gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:21:51.216023 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.215999    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:51.219424 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.219401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:51.225669 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.225645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config" pod "146fa115-70dd-4803-a27e-a8afda6b3bc7" (UID: "146fa115-70dd-4803-a27e-a8afda6b3bc7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:21:51.314484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314441    4528 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-out\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.314484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314477    4528 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-config-volume\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.314484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314487    4528 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-web-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.314484 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314496    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9gf\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-kube-api-access-lt9gf\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.314777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314504    4528 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/146fa115-70dd-4803-a27e-a8afda6b3bc7-cluster-tls-config\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.314777 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.314513    4528 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/146fa115-70dd-4803-a27e-a8afda6b3bc7-tls-assets\") on node \"ip-10-0-134-164.ec2.internal\" DevicePath \"\""
Oct 30 04:21:51.744791 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.744755    4528 generic.go:356] "Generic (PLEG): container finished" podID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerID="0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc" exitCode=0
Oct 30 04:21:51.744990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.744842    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"
Oct 30 04:21:51.744990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.744848    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerDied","Data":"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"}
Oct 30 04:21:51.744990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.744936    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"146fa115-70dd-4803-a27e-a8afda6b3bc7","Type":"ContainerDied","Data":"615aff531173044f5133e2b2a2afd073c8a38072c9f768129ae33682aebbabb8"}
Oct 30 04:21:51.744990 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.744967    4528 scope.go:122] "RemoveContainer" containerID="35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"
Oct 30 04:21:51.753543 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.753525    4528 scope.go:122] "RemoveContainer" containerID="0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"
Oct 30 04:21:51.761214 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.761196    4528 scope.go:122] "RemoveContainer" containerID="1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"
Oct 30 04:21:51.768461 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.768443    4528 scope.go:122] "RemoveContainer" containerID="35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"
Oct 30 04:21:51.768690 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:51.768675    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7\": container with ID starting with 35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7 not found: ID does not exist" containerID="35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"
Oct 30 04:21:51.768734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.768698    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7"} err="failed to get container status \"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7\": rpc error: code = NotFound desc = could not find container \"35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7\": container with ID starting with 35a4d6360d00395ddd22d1a36e8a318d8a607bda74bec0d5f0f7dcefb84c2ac7 not found: ID does not exist"
Oct 30 04:21:51.768734 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.768714    4528 scope.go:122] "RemoveContainer" containerID="0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"
Oct 30 04:21:51.768978 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:51.768956    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc\": container with ID starting with 0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc not found: ID does not exist" containerID="0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"
Oct 30 04:21:51.769025 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.768984    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc"} err="failed to get container status \"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc\": rpc error: code = NotFound desc = could not find container \"0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc\": container with ID starting with 0657d68f64e59158867e7d221307609aa69d60197eba43f5ee4886d17962dabc not found: ID does not exist"
Oct 30 04:21:51.769025 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.769001    4528 scope.go:122] "RemoveContainer" containerID="1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"
Oct 30 04:21:51.769260 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:21:51.769244    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18\": container with ID starting with 1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18 not found: ID does not exist" containerID="1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"
Oct 30 04:21:51.769313 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.769263    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18"} err="failed to get container status \"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18\": rpc error: code = NotFound desc = could not find container \"1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18\": container with ID starting with 1385df8b3216df2418f5166dfa6d7af7c8e88bc97f248ad9acc957dbecea9d18 not found: ID does not exist"
Oct 30 04:21:51.796556 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.796534    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:51.806327 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.806304    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"]
Oct 30 04:21:51.892981 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:51.892938    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" path="/var/lib/kubelet/pods/146fa115-70dd-4803-a27e-a8afda6b3bc7/volumes"
Oct 30 04:21:55.806514 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:55.806433    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:21:55.918752 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:21:55.918713    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:22:00.229185 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:00.229148    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j4z9f_d80631ef-63c9-400c-a46f-6eed3ca27590/global-pull-secret-syncer/0.log"
Oct 30 04:22:00.344889 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:00.344854    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log" limit=1024
Oct 30 04:22:01.000779 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:01.000745    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kgg59_163e0310-6238-4518-9d5a-c297c56f3603/konnectivity-agent/0.log"
Oct 30 04:22:01.420320 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:01.420280    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-164.ec2.internal_57d17cb448486bb943d1114ac107c4da/haproxy/0.log"
Oct 30 04:22:04.631603 ip-10-0-134-164 kubenswrapper[4528]: E1030 04:22:04.631565    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2hqz8" containerName="extractor"
Oct 30 04:22:07.834629 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:07.834598    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9r8z9_c6eb06d5-d557-4d8b-b61e-4c28f8140543/node-exporter/0.log"
Oct 30 04:22:07.955181 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:07.955144    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9r8z9_c6eb06d5-d557-4d8b-b61e-4c28f8140543/kube-rbac-proxy/0.log"
Oct 30 04:22:08.082510 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:08.082481    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9r8z9_c6eb06d5-d557-4d8b-b61e-4c28f8140543/init-textfile/0.log"
Oct 30 04:22:09.490957 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.490916    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"]
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491233    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b394e481-f889-4a02-997b-c29e2278c32e" containerName="otc-container"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491246    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b394e481-f889-4a02-997b-c29e2278c32e" containerName="otc-container"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491262    4528 cpu_manager.go:414] "containerMap: removing container" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491267    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491276    4528 cpu_manager.go:414] "containerMap: removing container" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491284    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491296    4528 cpu_manager.go:414] "containerMap: removing container" podUID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" containerName="thanos-querier"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491301    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" containerName="thanos-querier"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491310    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="init-config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491314    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="init-config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491320    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491324    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491334    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="prometheus"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491339    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="prometheus"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491346    4528 cpu_manager.go:414] "containerMap: removing container" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="init-config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491352    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="init-config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491366    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="thanos-sidecar"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491370    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="thanos-sidecar"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491415    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="191dfbf2-08c7-4ed9-89e0-76f33931c2a6" containerName="thanos-querier"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491427    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="config-reloader"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491436    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="prometheus"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491444    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4687d6d6-88a9-474b-bcfe-8182f4282a4f" containerName="thanos-sidecar"
Oct 30 04:22:09.491441 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491455    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b394e481-f889-4a02-997b-c29e2278c32e" containerName="otc-container"
Oct 30 04:22:09.492330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491466    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="alertmanager"
Oct 30 04:22:09.492330 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.491474    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="146fa115-70dd-4803-a27e-a8afda6b3bc7" containerName="config-reloader"
Oct 30 04:22:09.494223 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.494208    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.507094 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.507074    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zrscj\"/\"default-dockercfg-c8v22\""
Oct 30 04:22:09.516967 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.516941    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"openshift-service-ca.crt\""
Oct 30 04:22:09.517087 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.517045    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zrscj\"/\"kube-root-ca.crt\""
Oct 30 04:22:09.518548 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.518521    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"]
Oct 30 04:22:09.558017 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.557980    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-sys\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.558191 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.558021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw6k\" (UniqueName: \"kubernetes.io/projected/1f45e556-a9f6-4389-846a-c90d38e74290-kube-api-access-xpw6k\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.558191 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.558112    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-proc\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.558191 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.558151    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-podres\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.558191 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.558182    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-lib-modules\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659240 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659195    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-proc\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659249    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-podres\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-lib-modules\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659332    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-proc\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-sys\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659382    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-podres\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659397    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw6k\" (UniqueName: \"kubernetes.io/projected/1f45e556-a9f6-4389-846a-c90d38e74290-kube-api-access-xpw6k\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659450 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659412    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-sys\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.659707 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.659455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f45e556-a9f6-4389-846a-c90d38e74290-lib-modules\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.681620 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.681590    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw6k\" (UniqueName: \"kubernetes.io/projected/1f45e556-a9f6-4389-846a-c90d38e74290-kube-api-access-xpw6k\") pod \"perf-node-gather-daemonset-vnb5h\" (UID: \"1f45e556-a9f6-4389-846a-c90d38e74290\") " pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:09.807518 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:09.807432    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:10.101291 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.101195    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"]
Oct 30 04:22:10.106500 ip-10-0-134-164 kubenswrapper[4528]: W1030 04:22:10.106466    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f45e556_a9f6_4389_846a_c90d38e74290.slice/crio-d509ac936e312b64f881ddfa32f5f78abf975a2b802f22ace95a8164355c0add WatchSource:0}: Error finding container d509ac936e312b64f881ddfa32f5f78abf975a2b802f22ace95a8164355c0add: Status 404 returned error can't find the container with id d509ac936e312b64f881ddfa32f5f78abf975a2b802f22ace95a8164355c0add
Oct 30 04:22:10.208116 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.208085    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-szpz7_cd87f820-c610-4549-b9ea-235e07dd0810/prometheus-operator/0.log"
Oct 30 04:22:10.318920 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.318890    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-szpz7_cd87f820-c610-4549-b9ea-235e07dd0810/kube-rbac-proxy/0.log"
Oct 30 04:22:10.450552 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.450523    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5g6sc_adf40128-5210-4cf0-a318-14202b488553/prometheus-operator-admission-webhook/0.log"
Oct 30 04:22:10.832215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.832120    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h" event={"ID":"1f45e556-a9f6-4389-846a-c90d38e74290","Type":"ContainerStarted","Data":"d8bf0039ad20c92ec5fa261ce13ccbd6fc43d84343047c0b6fda7cae3ffc755b"}
Oct 30 04:22:10.832215 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.832190    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h" event={"ID":"1f45e556-a9f6-4389-846a-c90d38e74290","Type":"ContainerStarted","Data":"d509ac936e312b64f881ddfa32f5f78abf975a2b802f22ace95a8164355c0add"}
Oct 30 04:22:10.832618 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.832236    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:10.868120 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:10.868073    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h" podStartSLOduration=1.868059693 podStartE2EDuration="1.868059693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:10.867445257 +0000 UTC m=+6089.091694534" watchObservedRunningTime="2026-10-30 04:22:10.868059693 +0000 UTC m=+6089.092308964"
Oct 30 04:22:14.579151 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:14.579116    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5f97b844-qh55q_3c4f1c19-75e8-4985-ac23-5798150dc419/console/0.log"
Oct 30 04:22:14.727201 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:14.727171    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-vg56v_74047d51-f1c2-486d-814d-66b8da177110/download-server/0.log"
Oct 30 04:22:16.846293 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:16.846261    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zrscj/perf-node-gather-daemonset-vnb5h"
Oct 30 04:22:17.863529 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:17.863498    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7kjhk_84fedff4-406d-4691-b110-e7b297296f71/dns/0.log"
Oct 30 04:22:17.977860 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:17.977812    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7kjhk_84fedff4-406d-4691-b110-e7b297296f71/kube-rbac-proxy/0.log"
Oct 30 04:22:18.385080 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:18.385046    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2gm47_e75e3379-3433-4e3f-a6a0-52afc2f47330/dns-node-resolver/0.log"
Oct 30 04:22:19.730375 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:19.730334    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54fc56d44b-zq9ss_d77c483c-9cdd-4c94-b087-27c8bc611434/registry/0.log"
Oct 30 04:22:19.833782 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:19.833753    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nrwb4_5f7bd4f3-33ca-4c0f-9622-b45ecc97dc3b/node-ca/0.log"
Oct 30 04:22:22.571738 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:22.571707    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2cr8_5a135dd3-ea50-4c74-86d7-38c5e2d36ca9/serve-healthcheck-canary/0.log"
Oct 30 04:22:23.768704 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:23.768675    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hqz8_3b7e173f-fd22-4319-a168-d22dd2d3ca3b/kube-rbac-proxy/0.log"
Oct 30 04:22:23.885060 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:23.885029    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hqz8_3b7e173f-fd22-4319-a168-d22dd2d3ca3b/exporter/0.log"
Oct 30 04:22:24.001573 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:24.001543    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hqz8_3b7e173f-fd22-4319-a168-d22dd2d3ca3b/extractor/0.log"
Oct 30 04:22:24.851331 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:24.851303    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-6fc2t-6gzkc_1561e641-f0b5-4427-8b52-9dcddd81cc3d/insights-gathering/0.log"
Oct 30 04:22:24.972251 ip-10-0-134-164 kubenswrapper[4528]: I1030 04:22:24.972217    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-j7nq4-qbr4g_43c945bb-127b-4ce1-9807-dc3de3cb297b/insights-gathering/0.log"